Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
51 out of 60 is failed (threshold=6) [XX.XXXXXXXXXXXXXXX.XXXXXXX.X.XX.XXX.XX.XXXXXXX.XX.XXXXXXXXXX] duration:
5.001688388s trace:
https://jaeger.dev.storj.io/trace/3844b3a2f2557363
FAIL (2025-12-29 13:15:34 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001744805s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
31 out of 60 is failed (threshold=3) [X....XXXX..XX..X...XX.X.XX...X.X.X.XXX.X.X.X.XX..X.XXX.XX..X] duration:
30.004092351s trace:
https://jaeger.dev.storj.io/trace/4e9cc163fa850803
FAIL (2025-12-29 13:15:28 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: 4TWHAVGGSGRKTKDQTEIQVC3VXOOFYYI3LKEGFNIWML4U2UCSHBDA) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: 32T4H5B3CRKUA25W7EOTIUEERZ3D6LSOJ5GASLULAVDKLEQEXYTQ) context deadline exceeded
======LOGS======
Dec 29 13:14:59.052691585 (573h45m58.92377462s) [upload=503016]: Starting upload
Dec 29 13:14:59.052744516 (573h45m58.923827551s) [upload=503016]: Done starting upload
Dec 29 13:14:59.052915451 (573h45m58.923998496s) [upload=503016]: Uploading segments...
Dec 29 13:14:59.052954847 (573h45m58.924037882s) [upload=503016]: Got next segment. Inline: false
Dec 29 13:14:59.053000473 (573h45m58.924083508s) [upload=503016]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 13:14:59.053007426 (573h45m58.924090461s) [upload=503016]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 13:14:59.334293867 (573h45m59.205376912s) [seg_pos={0 0}, upload=503016]: Begin upload segment...
Dec 29 13:14:59.334347518 (573h45m59.205430563s) [seg_pos={0 0}, upload=503016]: Done begin upload segment.
Dec 29 13:14:59.334378097 (573h45m59.205461132s) [noise=true , address=145.79.227.18:4007 , piece_id=7E3ITPKMZPM4Y5FHVCGOEGHERB34CMAL2ES3TZ6SB3S3CHETRNBA, seg_pos={0 0}, upload=503016]: Uploading piece...
Dec 29 13:14:59.334400440 (573h45m59.205483485s) [noise=true , address=145.79.227.2:4114 , piece_id=NGIVG57VGFEFYWCRKPDNI42UFERFVBFZK6JOWPTUQNROMW2XUB2A, seg_pos={0 0}, upload=503016]: Uploading piece...
Dec 29 13:14:59.335194562 (573h45m59.206277597s) [noise=true , address=145.79.227.52:4445 , piece_id=R2PDEEEOJOVC4YNYKCPI7YI6B6M6YTCDC43ZF3MCY67B47X23AGQ, seg_pos={0 0}, upload=503016]: Uploading piece...
Dec 29 13:14:59.335784075 (573h45m59.20686712s) [noise=true , address=145.79.227.42:4122 , piece_id=2ZJBOZCTH3AYIPS7KOGGKQKNS436D3UNB55JSCDZ5PYC3RHHTGLQ, seg_pos={0 0}, upload=503016]: Uploading piece...
Dec 29 13:14:59.336178325 (573h45m59.20726137s) [noise=true , address=45.91.116.10:4084 , piece_id=4TWHAVGGSGRKTKDQTEIQVC3VXOOFYYI3LKEGFNIWML4U2UCSHBDA, seg_pos={0 0}, upload=503016]: Uploading piece...
Dec 29 13:14:59.336601731 (573h45m59.207684776s) [noise=true , address=145.79.227.12:4765 , piece_id=UPBPU2QVDRTTM5SWJMYIDL2W35GZFXAKEGFPY2YIDI7BQ55ARG6Q, seg_pos={0 0}, upload=503016]: Uploading piece...
Dec 29 13:14:59.337009347 (573h45m59.208092383s) [noise=true , address=145.79.227.11:4732 , piece_id=IG6W4GXSF2MINKPLWLTSFDWYIR7HMUODH26H6OHY2GTK6AYDV5NQ, seg_pos={0 0}, upload=503016]: Uploading piece...
Dec 29 13:14:59.337451069 (573h45m59.208534114s) [noise=true , address=77.90.31.16:4736 , piece_id=UV57UQIOHTCZJZTZMSM622PGJOPNBTE7L7MNOOCX355GHS2W6I2Q, seg_pos={0 0}, upload=503016]: Uploading piece...
Dec 29 13:14:59.337986047 (573h45m59.209069082s) [noise=true , address=77.90.31.12:4589 , piece_id=KFFXI3ZILAJTJKSUZ3TLOLC2JXYWWFG7IYBUUQSNPQOFUIC74PSA, seg_pos={0 0}, upload=503016]: Uploading piece...
Dec 29 13:14:59.338221566 (573h45m59.209304611s) [noise=true , address=145.79.227.13:4802 , piece_id=AZ7RK2VDHZYDYFRWHNHW6FMZJJZ3XRNOHN62EGVDNGBKBZ7J6SDQ, seg_pos={0 0}, upload=503016]: Uploading piece...
Dec 29 13:14:59.338678055 (573h45m59.2097611s) [noise=true , address=145.79.227.19:4077 , piece_id=BI7V27VENV4XRS4RYUGNHUEDWCV4UW2ZHBG367MJXHRUEPYOL5MA, seg_pos={0 0}, upload=503016]: Uploading piece...
Dec 29 13:14:59.338856685 (573h45m59.20993973s) [noise=true , address=45.91.116.11:4120 , piece_id=VKMMVQVYAGLKQGNVPH6G7G62HDMNIY3KDDHCDBDBY6MG4YFDJPWQ, seg_pos={0 0}, upload=503016]: Uploading piece...
Dec 29 13:14:59.339166555 (573h45m59.21024959s) [noise=true , address=145.79.227.32:4834 , piece_id=V5TUJNYHCLJXQVEVOTZDXIXJRFEQO3CPOTUU5KDHPNPFSOLURF4Q, seg_pos={0 0}, upload=503016]: Uploading piece...
Dec 29 13:14:59.339443873 (573h45m59.210526918s) [noise=true , address=145.79.227.38:4979 , piece_id=NK5USLTPICTEVKDWIBAL735G73RE2HBDGOTULXFG2NQRQDYKBVWA, seg_pos={0 0}, upload=503016]: Uploading piece...
Dec 29 13:14:59.340070727 (573h45m59.211153782s) [noise=true , address=77.90.31.11:4556 , piece_id=32T4H5B3CRKUA25W7EOTIUEERZ3D6LSOJ5GASLULAVDKLEQEXYTQ, seg_pos={0 0}, upload=503016]: Uploading piece...
Dec 29 13:14:59.340738428 (573h45m59.211821463s) [noise=true , address=145.79.227.16:4912 , piece_id=JMH2XIRXCBLL6BY7NL2KI7MZXWPEDC6MAPOPORWZXB55XJJAPXHA, seg_pos={0 0}, upload=503016]: Uploading piece...
Dec 29 13:15:00.997623811 (573h46m0.868706856s) [upload=503016]: Next returned nil segment
Dec 29 13:15:00.997646735 (573h46m0.86872979s) [upload=503016]: Waiting for error group managing segments...
Dec 29 13:15:01.322861973 (573h46m1.193945028s) [noise=true , address=145.79.227.38:4979 , piece_id=NK5USLTPICTEVKDWIBAL735G73RE2HBDGOTULXFG2NQRQDYKBVWA, seg_pos={0 0}, upload=503016]: Done uploading piece. err: <nil>
Dec 29 13:15:01.323060331 (573h46m1.194143376s) [noise=true , address=145.79.227.13:4802 , piece_id=AZ7RK2VDHZYDYFRWHNHW6FMZJJZ3XRNOHN62EGVDNGBKBZ7J6SDQ, seg_pos={0 0}, upload=503016]: Done uploading piece. err: <nil>
Dec 29 13:15:01.329368823 (573h46m1.200451868s) [noise=true , address=145.79.227.18:4007 , piece_id=7E3ITPKMZPM4Y5FHVCGOEGHERB34CMAL2ES3TZ6SB3S3CHETRNBA, seg_pos={0 0}, upload=503016]: Done uploading piece. err: <nil>
Dec 29 13:15:01.330298153 (573h46m1.201381188s) [noise=true , address=145.79.227.32:4834 , piece_id=V5TUJNYHCLJXQVEVOTZDXIXJRFEQO3CPOTUU5KDHPNPFSOLURF4Q, seg_pos={0 0}, upload=503016]: Done uploading piece. err: <nil>
Dec 29 13:15:01.331235036 (573h46m1.202318091s) [noise=true , address=145.79.227.19:4077 , piece_id=BI7V27VENV4XRS4RYUGNHUEDWCV4UW2ZHBG367MJXHRUEPYOL5MA, seg_pos={0 0}, upload=503016]: Done uploading piece. err: <nil>
Dec 29 13:15:01.333910471 (573h46m1.204993516s) [noise=true , address=145.79.227.52:4445 , piece_id=R2PDEEEOJOVC4YNYKCPI7YI6B6M6YTCDC43ZF3MCY67B47X23AGQ, seg_pos={0 0}, upload=503016]: Done uploading piece. err: <nil>
Dec 29 13:15:01.334003288 (573h46m1.205086343s) [noise=true , address=145.79.227.12:4765 , piece_id=UPBPU2QVDRTTM5SWJMYIDL2W35GZFXAKEGFPY2YIDI7BQ55ARG6Q, seg_pos={0 0}, upload=503016]: Done uploading piece. err: <nil>
Dec 29 13:15:01.344178402 (573h46m1.215261448s) [noise=true , address=145.79.227.16:4912 , piece_id=JMH2XIRXCBLL6BY7NL2KI7MZXWPEDC6MAPOPORWZXB55XJJAPXHA, seg_pos={0 0}, upload=503016]: Done uploading piece. err: <nil>
Dec 29 13:15:01.347340464 (573h46m1.218423509s) [noise=true , address=145.79.227.11:4732 , piece_id=IG6W4GXSF2MINKPLWLTSFDWYIR7HMUODH26H6OHY2GTK6AYDV5NQ, seg_pos={0 0}, upload=503016]: Done uploading piece. err: <nil>
Dec 29 13:15:01.347892945 (573h46m1.21897599s) [noise=true , address=145.79.227.2:4114 , piece_id=NGIVG57VGFEFYWCRKPDNI42UFERFVBFZK6JOWPTUQNROMW2XUB2A, seg_pos={0 0}, upload=503016]: Done uploading piece. err: <nil>
Dec 29 13:15:01.349781852 (573h46m1.220864898s) [noise=true , address=145.79.227.42:4122 , piece_id=2ZJBOZCTH3AYIPS7KOGGKQKNS436D3UNB55JSCDZ5PYC3RHHTGLQ, seg_pos={0 0}, upload=503016]: Done uploading piece. err: <nil>
Dec 29 13:15:01.572830599 (573h46m1.443913644s) [noise=true , address=45.91.116.10:4084 , piece_id=4TWHAVGGSGRKTKDQTEIQVC3VXOOFYYI3LKEGFNIWML4U2UCSHBDA, seg_pos={0 0}, upload=503016]: Done uploading piece. err: <nil>
Dec 29 13:15:01.640741902 (573h46m1.511824957s) [noise=true , address=77.90.31.11:4556 , piece_id=32T4H5B3CRKUA25W7EOTIUEERZ3D6LSOJ5GASLULAVDKLEQEXYTQ, seg_pos={0 0}, upload=503016]: Done uploading piece. err: <nil>
Dec 29 13:15:01.640778241 (573h46m1.511861287s) [seg_pos={0 0}, upload=503016]: Segment reached optimal threshold of 13 pieces.
Dec 29 13:15:01.641000604 (573h46m1.51208365s) [noise=true , address=45.91.116.11:4120 , piece_id=VKMMVQVYAGLKQGNVPH6G7G62HDMNIY3KDDHCDBDBY6MG4YFDJPWQ, seg_pos={0 0}, upload=503016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF
Dec 29 13:15:01.641007858 (573h46m1.512090903s) [noise=true , address=45.91.116.11:4120 , piece_id=VKMMVQVYAGLKQGNVPH6G7G62HDMNIY3KDDHCDBDBY6MG4YFDJPWQ, seg_pos={0 0}, upload=503016]: Long tail cancellation due to optimal threshold
Dec 29 13:15:01.641330322 (573h46m1.512413367s) [noise=true , address=77.90.31.12:4589 , piece_id=KFFXI3ZILAJTJKSUZ3TLOLC2JXYWWFG7IYBUUQSNPQOFUIC74PSA, seg_pos={0 0}, upload=503016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF
Dec 29 13:15:01.641339240 (573h46m1.512422285s) [noise=true , address=77.90.31.12:4589 , piece_id=KFFXI3ZILAJTJKSUZ3TLOLC2JXYWWFG7IYBUUQSNPQOFUIC74PSA, seg_pos={0 0}, upload=503016]: Long tail cancellation due to optimal threshold
Dec 29 13:15:01.641560470 (573h46m1.512643516s) [noise=true , address=77.90.31.16:4736 , piece_id=UV57UQIOHTCZJZTZMSM622PGJOPNBTE7L7MNOOCX355GHS2W6I2Q, seg_pos={0 0}, upload=503016]: Done uploading piece. err: <nil>
Dec 29 13:15:01.641636635 (573h46m1.51271968s) [seg_pos={0 0}, upload=503016]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil>
Dec 29 13:15:01.641752587 (573h46m1.512835632s) [upload=503016]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:15:01.641759810 (573h46m1.512842855s) [upload=503016]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:15:01.938778637 (573h46m1.809861682s) [upload=503016]: Done uploading segments...
Dec 29 13:15:01.938806460 (573h46m1.809889515s) [upload=503016]: Upload finished. err: <nil>
Duration: 30.004117038s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [....................X......................X....X.X....X.X..] duration:
14.371367612s trace:
https://jaeger.dev.storj.io/trace/312ddb07b2dee82a
FAIL (2025-12-29 13:15:15 UTC)
orbiter warning: context deadline exceeded
Duration: 14.371400396s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
51 out of 60 is failed (threshold=6) [XXX.XXXXXXXXXXXXXXX.XXXXXXX.X.XX.XXX.XX.XXXXXXX.XX.XXXXXXXXX] duration:
5.002713205s trace:
https://jaeger.dev.storj.io/trace/2f8a4abdce56b1b5
FAIL (2025-12-29 13:15:05 UTC)
orbiter warning: context deadline exceeded
Duration: 5.002738764s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [....X......................X.....X.....................XX...] duration:
48.53116564s trace:
https://jaeger.dev.storj.io/trace/77d26141ea11d228
FAIL (2025-12-29 13:15:00 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu, Piece ID: JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ) context deadline exceeded; piecestore: (Node ID: 125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2, Piece ID: INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA) context deadline exceeded; piecestore: (Node ID: 12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL, Piece ID: Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA) context deadline exceeded
======LOGS======
Q2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.692912334 (573h43m3.563995379s) [noise=true , address=77.90.31.16:4759 , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.693322595 (573h43m3.56440564s) [noise=true , address=77.90.31.15:4711 , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.693739649 (573h43m3.564822694s) [noise=true , address=45.91.116.6:4981 , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.694315887 (573h43m3.565398942s) [noise=true , address=145.79.227.4:4220 , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.694765222 (573h43m3.565848267s) [noise=true , address=145.79.227.29:4952 , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.695202545 (573h43m3.56628559s) [noise=true , address=145.79.227.40:4058 , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.695687528 (573h43m3.566770584s) [noise=true , address=77.90.31.19:4857 , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.696135221 (573h43m3.567218276s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.696703733 (573h43m3.567786778s) [noise=true , address=64.49.13.170:4720 , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.698418298 (573h43m3.569501343s) [noise=true , address=145.79.227.36:4520 , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.700198829 (573h43m3.571281874s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.701993145 (573h43m3.57307619s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.703884687 (573h43m3.574967732s) [noise=true , address=145.79.227.33:4893 , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.705687109 (573h43m3.576770154s) [noise=true , address=45.91.116.10:4094 , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.707557651 (573h43m3.578640696s) [noise=true , address=145.79.227.43:4164 , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:05.168247719 (573h43m5.039330754s) [upload=502975]: Next returned nil segment
Dec 29 13:12:05.168259962 (573h43m5.039343007s) [upload=502975]: Waiting for error group managing segments...
Dec 29 13:12:05.558873127 (573h43m5.429956162s) [noise=true , address=145.79.227.4:4220 , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.563024312 (573h43m5.434107347s) [noise=true , address=64.49.13.170:4720 , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.564115719 (573h43m5.435198755s) [noise=true , address=145.79.227.36:4520 , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.593873402 (573h43m5.464956447s) [noise=true , address=145.79.227.29:4952 , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.898194604 (573h43m5.769277639s) [noise=true , address=45.91.116.10:4094 , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.912845292 (573h43m5.783928337s) [noise=true , address=77.90.31.19:4857 , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.426532207 (573h43m6.297615252s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.449589562 (573h43m6.320672607s) [noise=true , address=77.90.31.15:4711 , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.453186841 (573h43m6.324269876s) [noise=true , address=145.79.227.40:4058 , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.510769276 (573h43m6.381852321s) [noise=true , address=145.79.227.43:4164 , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.518868879 (573h43m6.389951924s) [noise=true , address=145.79.227.33:4893 , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.519238043 (573h43m6.390321098s) [noise=true , address=45.91.116.6:4981 , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.523211328 (573h43m6.394294373s) [noise=true , address=77.90.31.16:4759 , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.767869783 (573h43m6.638952818s) [noise=true , address=145.79.227.34:4924 , piece_id=MAQXQTAGXQ7AEPUCESUSUGMHSDMMXS7HHKAEA7B3HW5X66PDUF5A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.767889431 (573h43m6.638972476s) [seg_pos={0 0}, upload=502975]: Segment reached optimal threshold of 14 pieces.
Dec 29 13:12:06.768102116 (573h43m6.639185161s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:06.768108257 (573h43m6.639191302s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768219208 (573h43m6.639302254s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:06.768224789 (573h43m6.639307824s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768384213 (573h43m6.639467258s) [noise=true , address=64.49.13.169:4711 , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection
Dec 29 13:12:06.768399261 (573h43m6.639482296s) [noise=true , address=64.49.13.169:4711 , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768732636 (573h43m6.639815671s) [noise=true , address=64.49.13.169:4709 , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection
Dec 29 13:12:06.768745460 (573h43m6.639828505s) [noise=true , address=64.49.13.169:4709 , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768784586 (573h43m6.639867631s) [seg_pos={0 0}, upload=502975]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 13:12:06.768880217 (573h43m6.639963252s) [upload=502975]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:12:06.768884906 (573h43m6.639967941s) [upload=502975]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:12:07.067188330 (573h43m6.938271365s) [upload=502975]: Done uploading segments...
Dec 29 13:12:07.067251099 (573h43m6.938334134s) [upload=502975]: Upload finished. err: <nil>
Duration: 48.531200095s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
51 out of 60 is failed (threshold=6) [XXXX.XXXXXXXXXXXXXXX.XXXXXXX.X.XX.XXX.XX.XXXXXXX.XX.XXXXXXXX] duration:
5.001349691s trace:
https://jaeger.dev.storj.io/trace/2981da047f45a9c0
FAIL (2025-12-29 13:14:42 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001368786s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.....................X......................X....X.X....X.X.] duration:
24.338901552s trace:
https://jaeger.dev.storj.io/trace/312ddb07b2dee82a
FAIL (2025-12-29 13:14:40 UTC)
orbiter warning: context deadline exceeded
Duration: 24.338935447s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.....X......................X.....X.....................XX..] duration:
37.372482362s trace:
https://jaeger.dev.storj.io/trace/77d26141ea11d228
FAIL (2025-12-29 13:14:10 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu, Piece ID: JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ) context deadline exceeded; piecestore: (Node ID: 125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2, Piece ID: INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA) context deadline exceeded; piecestore: (Node ID: 12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL, Piece ID: Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA) context deadline exceeded
======LOGS======
Q2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.692912334 (573h43m3.563995379s) [noise=true , address=77.90.31.16:4759 , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.693322595 (573h43m3.56440564s) [noise=true , address=77.90.31.15:4711 , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.693739649 (573h43m3.564822694s) [noise=true , address=45.91.116.6:4981 , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.694315887 (573h43m3.565398942s) [noise=true , address=145.79.227.4:4220 , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.694765222 (573h43m3.565848267s) [noise=true , address=145.79.227.29:4952 , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.695202545 (573h43m3.56628559s) [noise=true , address=145.79.227.40:4058 , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.695687528 (573h43m3.566770584s) [noise=true , address=77.90.31.19:4857 , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.696135221 (573h43m3.567218276s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.696703733 (573h43m3.567786778s) [noise=true , address=64.49.13.170:4720 , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.698418298 (573h43m3.569501343s) [noise=true , address=145.79.227.36:4520 , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.700198829 (573h43m3.571281874s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.701993145 (573h43m3.57307619s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.703884687 (573h43m3.574967732s) [noise=true , address=145.79.227.33:4893 , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.705687109 (573h43m3.576770154s) [noise=true , address=45.91.116.10:4094 , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.707557651 (573h43m3.578640696s) [noise=true , address=145.79.227.43:4164 , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:05.168247719 (573h43m5.039330754s) [upload=502975]: Next returned nil segment
Dec 29 13:12:05.168259962 (573h43m5.039343007s) [upload=502975]: Waiting for error group managing segments...
Dec 29 13:12:05.558873127 (573h43m5.429956162s) [noise=true , address=145.79.227.4:4220 , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.563024312 (573h43m5.434107347s) [noise=true , address=64.49.13.170:4720 , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.564115719 (573h43m5.435198755s) [noise=true , address=145.79.227.36:4520 , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.593873402 (573h43m5.464956447s) [noise=true , address=145.79.227.29:4952 , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.898194604 (573h43m5.769277639s) [noise=true , address=45.91.116.10:4094 , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.912845292 (573h43m5.783928337s) [noise=true , address=77.90.31.19:4857 , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.426532207 (573h43m6.297615252s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.449589562 (573h43m6.320672607s) [noise=true , address=77.90.31.15:4711 , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.453186841 (573h43m6.324269876s) [noise=true , address=145.79.227.40:4058 , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.510769276 (573h43m6.381852321s) [noise=true , address=145.79.227.43:4164 , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.518868879 (573h43m6.389951924s) [noise=true , address=145.79.227.33:4893 , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.519238043 (573h43m6.390321098s) [noise=true , address=45.91.116.6:4981 , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.523211328 (573h43m6.394294373s) [noise=true , address=77.90.31.16:4759 , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.767869783 (573h43m6.638952818s) [noise=true , address=145.79.227.34:4924 , piece_id=MAQXQTAGXQ7AEPUCESUSUGMHSDMMXS7HHKAEA7B3HW5X66PDUF5A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.767889431 (573h43m6.638972476s) [seg_pos={0 0}, upload=502975]: Segment reached optimal threshold of 14 pieces.
Dec 29 13:12:06.768102116 (573h43m6.639185161s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:06.768108257 (573h43m6.639191302s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768219208 (573h43m6.639302254s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:06.768224789 (573h43m6.639307824s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768384213 (573h43m6.639467258s) [noise=true , address=64.49.13.169:4711 , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection
Dec 29 13:12:06.768399261 (573h43m6.639482296s) [noise=true , address=64.49.13.169:4711 , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768732636 (573h43m6.639815671s) [noise=true , address=64.49.13.169:4709 , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection
Dec 29 13:12:06.768745460 (573h43m6.639828505s) [noise=true , address=64.49.13.169:4709 , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768784586 (573h43m6.639867631s) [seg_pos={0 0}, upload=502975]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 13:12:06.768880217 (573h43m6.639963252s) [upload=502975]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:12:06.768884906 (573h43m6.639967941s) [upload=502975]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:12:07.067188330 (573h43m6.938271365s) [upload=502975]: Done uploading segments...
Dec 29 13:12:07.067251099 (573h43m6.938334134s) [upload=502975]: Upload finished. err: <nil>
Duration: 37.372516428s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
30 out of 60 is failed (threshold=3) [.X....XXXX..XX..X...XX.X.XX...X.X.X.XXX.X.X.X.XX..X.XXX.XX..] duration:
6.005334813s trace:
https://jaeger.dev.storj.io/trace/4d0a30f64135bb3
FAIL (2025-12-29 13:14:00 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 29 13:12:04.985229816 (573h43m4.856312861s) [upload=502976]: Starting upload
Dec 29 13:12:04.985377688 (573h43m4.856460743s) [upload=502976]: Done starting upload
Dec 29 13:12:04.985451167 (573h43m4.856534243s) [upload=502976]: Uploading segments...
Dec 29 13:12:04.985745158 (573h43m4.856828193s) [upload=502976]: Got next segment. Inline: false
Dec 29 13:12:04.985790995 (573h43m4.85687404s) [upload=502976]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 13:12:04.985825440 (573h43m4.856908495s) [upload=502976]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 13:12:05.233743983 (573h43m5.104827028s) [seg_pos={0 0}, upload=502976]: Begin upload segment...
Dec 29 13:12:05.233879912 (573h43m5.104962957s) [noise=true , address=145.79.227.10:4401 , piece_id=DVRBJ3ZFYNR2GNTSM7ZBPZZKGSDDWR4XCFJ72VRBXBE2ZDMSFYYQ, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:05.233976326 (573h43m5.105059381s) [seg_pos={0 0}, upload=502976]: Done begin upload segment.
Dec 29 13:12:05.234045397 (573h43m5.105128432s) [noise=true , address=145.79.227.37:4566 , piece_id=ISMGHRTBVSAHHRKNV2PJV7WDAEKYPT6YLXAV4VBL7VEAEHCLUABQ, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:05.234761752 (573h43m5.105844787s) [noise=true , address=77.90.31.19:4844 , piece_id=CX7T4OCPGQIPVZ7PQG7YAC56DG5XPIGPBVOJUOV7KD77N6ZHNS4A, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:05.235202040 (573h43m5.106285075s) [noise=true , address=77.90.31.17:4773 , piece_id=MU5NJS6HG5X2ZZLUJJROZEWIU2RPDTZMHCTU6HE2QW3VGRORPZWQ, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:05.235697533 (573h43m5.106780578s) [noise=true , address=145.79.227.38:4979 , piece_id=WV7YS3MGMD2KYAA3ORTOS6YGK3KQZTUG4XKFNQXVOOH4THF5WCRA, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:05.236310140 (573h43m5.107393185s) [noise=true , address=145.79.227.20:4437 , piece_id=AX7WIS6EWJWZBPQ7JAPCUNDVZEL64FDM5GOVEK6V4QPKFORYO6CA, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:05.236905073 (573h43m5.107988108s) [noise=true , address=45.91.116.6:4976 , piece_id=ONS6LPVAFX2BKHENMZAZZID3WZOPQV7CJ2NU3VG2W2LHB2SVISNA, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:05.244302288 (573h43m5.115385333s) [noise=true , address=145.79.227.11:4732 , piece_id=FUDW7B6RL3MSTOTUAOFNGFV4OSPZUIVW574H5KP7W7CUAGXKZP4Q, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:05.245288366 (573h43m5.116371401s) [noise=true , address=145.79.227.51:4410 , piece_id=3K57LINT2MCQEUYY3MQWW7ZMW7IHPTQLESIKW4KHJJYYXDKDOPZQ, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:05.247332598 (573h43m5.118415633s) [noise=true , address=145.79.227.15:4876 , piece_id=P62GW63T7PUKQJ2M5424CUMOXKMRH4J366Z5ULETXLRKDVT2D7FQ, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:05.248844728 (573h43m5.119927773s) [noise=true , address=145.79.227.19:4079 , piece_id=NUJALKQUNVJSHQUX5JPAJX3HGFFEXTYNVOH2PUKGF73ZFKDDQFHQ, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:05.249290426 (573h43m5.120373471s) [noise=true , address=145.79.227.35:4481 , piece_id=TDHAUE3DN3DCBXXI66NFKCXU63ZCAQJSZAYU56IMSGX5BQJUJFZQ, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:05.250997606 (573h43m5.122080641s) [noise=true , address=77.90.31.14:4661 , piece_id=4GB66L3GH5YEAZMYITBDPKQT3I6GKJ2KQXEZ77PUM3F3TRZ3RDYQ, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:05.251918220 (573h43m5.123001255s) [noise=true , address=145.79.227.18:4007 , piece_id=DY5QZOE7ZNROSR6SPCHUITETGKCZZ4ZGMIKHKULKU7JGIUXZ76DQ, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:05.253074702 (573h43m5.124157737s) [noise=true , address=45.91.116.10:4082 , piece_id=JRDRU2EYT2TFZFXVQG6RBGTQ4PADBQDGSJOQBFVCKXRIHH5SAT2Q, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:05.254594075 (573h43m5.12567711s) [noise=true , address=145.79.227.7:4293 , piece_id=FSJYLIH2RXZY5XJUTQTMUXR6PLGXTPXHSADBBRYTWCWK6DAIXJBA, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:06.939401444 (573h43m6.81048448s) [upload=502976]: Next returned nil segment
Dec 29 13:12:06.939422725 (573h43m6.81050576s) [upload=502976]: Waiting for error group managing segments...
Dec 29 13:12:07.219898942 (573h43m7.090981987s) [noise=true , address=145.79.227.18:4007 , piece_id=DY5QZOE7ZNROSR6SPCHUITETGKCZZ4ZGMIKHKULKU7JGIUXZ76DQ, seg_pos={0 0}, upload=502976]: Done uploading piece. err: <nil>
Dec 29 13:12:07.230318393 (573h43m7.101401429s) [noise=true , address=145.79.227.20:4437 , piece_id=AX7WIS6EWJWZBPQ7JAPCUNDVZEL64FDM5GOVEK6V4QPKFORYO6CA, seg_pos={0 0}, upload=502976]: Done uploading piece. err: <nil>
Dec 29 13:12:07.230346387 (573h43m7.101429422s) [noise=true , address=145.79.227.10:4401 , piece_id=DVRBJ3ZFYNR2GNTSM7ZBPZZKGSDDWR4XCFJ72VRBXBE2ZDMSFYYQ, seg_pos={0 0}, upload=502976]: Done uploading piece. err: <nil>
Dec 29 13:12:07.235223112 (573h43m7.106306157s) [noise=true , address=145.79.227.15:4876 , piece_id=P62GW63T7PUKQJ2M5424CUMOXKMRH4J366Z5ULETXLRKDVT2D7FQ, seg_pos={0 0}, upload=502976]: Done uploading piece. err: <nil>
Dec 29 13:12:07.235504668 (573h43m7.106587703s) [noise=true , address=145.79.227.38:4979 , piece_id=WV7YS3MGMD2KYAA3ORTOS6YGK3KQZTUG4XKFNQXVOOH4THF5WCRA, seg_pos={0 0}, upload=502976]: Done uploading piece. err: <nil>
Dec 29 13:12:07.236420412 (573h43m7.107503448s) [noise=true , address=145.79.227.51:4410 , piece_id=3K57LINT2MCQEUYY3MQWW7ZMW7IHPTQLESIKW4KHJJYYXDKDOPZQ, seg_pos={0 0}, upload=502976]: Done uploading piece. err: <nil>
Dec 29 13:12:07.252096952 (573h43m7.123180007s) [noise=true , address=145.79.227.37:4566 , piece_id=ISMGHRTBVSAHHRKNV2PJV7WDAEKYPT6YLXAV4VBL7VEAEHCLUABQ, seg_pos={0 0}, upload=502976]: Done uploading piece. err: <nil>
Dec 29 13:12:07.252314717 (573h43m7.123397762s) [noise=true , address=145.79.227.11:4732 , piece_id=FUDW7B6RL3MSTOTUAOFNGFV4OSPZUIVW574H5KP7W7CUAGXKZP4Q, seg_pos={0 0}, upload=502976]: Done uploading piece. err: <nil>
Dec 29 13:12:07.258573716 (573h43m7.129656751s) [noise=true , address=145.79.227.19:4079 , piece_id=NUJALKQUNVJSHQUX5JPAJX3HGFFEXTYNVOH2PUKGF73ZFKDDQFHQ, seg_pos={0 0}, upload=502976]: Done uploading piece. err: <nil>
Dec 29 13:12:07.265694585 (573h43m7.13677762s) [noise=true , address=145.79.227.7:4293 , piece_id=FSJYLIH2RXZY5XJUTQTMUXR6PLGXTPXHSADBBRYTWCWK6DAIXJBA, seg_pos={0 0}, upload=502976]: Done uploading piece. err: <nil>
Dec 29 13:12:07.882198598 (573h43m7.753281643s) [noise=true , address=45.91.116.10:4082 , piece_id=JRDRU2EYT2TFZFXVQG6RBGTQ4PADBQDGSJOQBFVCKXRIHH5SAT2Q, seg_pos={0 0}, upload=502976]: Done uploading piece. err: <nil>
Dec 29 13:12:08.193700432 (573h43m8.064783467s) [noise=true , address=77.90.31.17:4773 , piece_id=MU5NJS6HG5X2ZZLUJJROZEWIU2RPDTZMHCTU6HE2QW3VGRORPZWQ, seg_pos={0 0}, upload=502976]: Done uploading piece. err: <nil>
Dec 29 13:12:08.385481111 (573h43m8.256564146s) [noise=true , address=45.91.116.6:4976 , piece_id=ONS6LPVAFX2BKHENMZAZZID3WZOPQV7CJ2NU3VG2W2LHB2SVISNA, seg_pos={0 0}, upload=502976]: Done uploading piece. err: <nil>
Dec 29 13:12:08.385506859 (573h43m8.256589894s) [seg_pos={0 0}, upload=502976]: Segment reached optimal threshold of 13 pieces.
Dec 29 13:12:08.385722060 (573h43m8.256805095s) [noise=true , address=77.90.31.14:4661 , piece_id=4GB66L3GH5YEAZMYITBDPKQT3I6GKJ2KQXEZ77PUM3F3TRZ3RDYQ, seg_pos={0 0}, upload=502976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:08.385728011 (573h43m8.256811046s) [noise=true , address=77.90.31.14:4661 , piece_id=4GB66L3GH5YEAZMYITBDPKQT3I6GKJ2KQXEZ77PUM3F3TRZ3RDYQ, seg_pos={0 0}, upload=502976]: Long tail cancellation due to optimal threshold
Dec 29 13:12:08.385829484 (573h43m8.256912529s) [noise=true , address=77.90.31.19:4844 , piece_id=CX7T4OCPGQIPVZ7PQG7YAC56DG5XPIGPBVOJUOV7KD77N6ZHNS4A, seg_pos={0 0}, upload=502976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:08.385845464 (573h43m8.256928499s) [noise=true , address=77.90.31.19:4844 , piece_id=CX7T4OCPGQIPVZ7PQG7YAC56DG5XPIGPBVOJUOV7KD77N6ZHNS4A, seg_pos={0 0}, upload=502976]: Long tail cancellation due to optimal threshold
Dec 29 13:12:08.386004277 (573h43m8.257087312s) [noise=true , address=145.79.227.35:4481 , piece_id=TDHAUE3DN3DCBXXI66NFKCXU63ZCAQJSZAYU56IMSGX5BQJUJFZQ, seg_pos={0 0}, upload=502976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:08.386011150 (573h43m8.257094185s) [noise=true , address=145.79.227.35:4481 , piece_id=TDHAUE3DN3DCBXXI66NFKCXU63ZCAQJSZAYU56IMSGX5BQJUJFZQ, seg_pos={0 0}, upload=502976]: Long tail cancellation due to optimal threshold
Dec 29 13:12:08.386028823 (573h43m8.257111858s) [seg_pos={0 0}, upload=502976]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 13:12:08.386168599 (573h43m8.257251644s) [upload=502976]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:12:08.386176204 (573h43m8.257259249s) [upload=502976]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:12:08.691718291 (573h43m8.562801346s) [upload=502976]: Done uploading segments...
Dec 29 13:12:08.691772264 (573h43m8.562855299s) [upload=502976]: Upload finished. err: <nil>
Duration: 6.005362155s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [......................X......................X....X.X....X.X] duration:
30.001614821s trace:
https://jaeger.dev.storj.io/trace/312ddb07b2dee82a
FAIL (2025-12-29 13:13:51 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001639238s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
51 out of 60 is failed (threshold=6) [XXXXX.XXXXXXXXXXXXXXX.XXXXXXX.X.XX.XXX.XX.XXXXXXX.XX.XXXXXXX] duration:
5.000788907s trace:
https://jaeger.dev.storj.io/trace/1aef95b6db51ce34
FAIL (2025-12-29 13:13:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000812604s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [......X......................X.....X.....................XX.] duration:
15.361001196s trace:
https://jaeger.dev.storj.io/trace/77d26141ea11d228
FAIL (2025-12-29 13:13:18 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu, Piece ID: JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ) context deadline exceeded; piecestore: (Node ID: 125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2, Piece ID: INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA) context deadline exceeded; piecestore: (Node ID: 12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL, Piece ID: Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA) context deadline exceeded
======LOGS======
Q2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.692912334 (573h43m3.563995379s) [noise=true , address=77.90.31.16:4759 , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.693322595 (573h43m3.56440564s) [noise=true , address=77.90.31.15:4711 , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.693739649 (573h43m3.564822694s) [noise=true , address=45.91.116.6:4981 , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.694315887 (573h43m3.565398942s) [noise=true , address=145.79.227.4:4220 , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.694765222 (573h43m3.565848267s) [noise=true , address=145.79.227.29:4952 , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.695202545 (573h43m3.56628559s) [noise=true , address=145.79.227.40:4058 , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.695687528 (573h43m3.566770584s) [noise=true , address=77.90.31.19:4857 , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.696135221 (573h43m3.567218276s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.696703733 (573h43m3.567786778s) [noise=true , address=64.49.13.170:4720 , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.698418298 (573h43m3.569501343s) [noise=true , address=145.79.227.36:4520 , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.700198829 (573h43m3.571281874s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.701993145 (573h43m3.57307619s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.703884687 (573h43m3.574967732s) [noise=true , address=145.79.227.33:4893 , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.705687109 (573h43m3.576770154s) [noise=true , address=45.91.116.10:4094 , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.707557651 (573h43m3.578640696s) [noise=true , address=145.79.227.43:4164 , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:05.168247719 (573h43m5.039330754s) [upload=502975]: Next returned nil segment
Dec 29 13:12:05.168259962 (573h43m5.039343007s) [upload=502975]: Waiting for error group managing segments...
Dec 29 13:12:05.558873127 (573h43m5.429956162s) [noise=true , address=145.79.227.4:4220 , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.563024312 (573h43m5.434107347s) [noise=true , address=64.49.13.170:4720 , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.564115719 (573h43m5.435198755s) [noise=true , address=145.79.227.36:4520 , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.593873402 (573h43m5.464956447s) [noise=true , address=145.79.227.29:4952 , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.898194604 (573h43m5.769277639s) [noise=true , address=45.91.116.10:4094 , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.912845292 (573h43m5.783928337s) [noise=true , address=77.90.31.19:4857 , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.426532207 (573h43m6.297615252s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.449589562 (573h43m6.320672607s) [noise=true , address=77.90.31.15:4711 , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.453186841 (573h43m6.324269876s) [noise=true , address=145.79.227.40:4058 , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.510769276 (573h43m6.381852321s) [noise=true , address=145.79.227.43:4164 , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.518868879 (573h43m6.389951924s) [noise=true , address=145.79.227.33:4893 , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.519238043 (573h43m6.390321098s) [noise=true , address=45.91.116.6:4981 , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.523211328 (573h43m6.394294373s) [noise=true , address=77.90.31.16:4759 , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.767869783 (573h43m6.638952818s) [noise=true , address=145.79.227.34:4924 , piece_id=MAQXQTAGXQ7AEPUCESUSUGMHSDMMXS7HHKAEA7B3HW5X66PDUF5A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.767889431 (573h43m6.638972476s) [seg_pos={0 0}, upload=502975]: Segment reached optimal threshold of 14 pieces.
Dec 29 13:12:06.768102116 (573h43m6.639185161s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:06.768108257 (573h43m6.639191302s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768219208 (573h43m6.639302254s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:06.768224789 (573h43m6.639307824s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768384213 (573h43m6.639467258s) [noise=true , address=64.49.13.169:4711 , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection
Dec 29 13:12:06.768399261 (573h43m6.639482296s) [noise=true , address=64.49.13.169:4711 , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768732636 (573h43m6.639815671s) [noise=true , address=64.49.13.169:4709 , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection
Dec 29 13:12:06.768745460 (573h43m6.639828505s) [noise=true , address=64.49.13.169:4709 , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768784586 (573h43m6.639867631s) [seg_pos={0 0}, upload=502975]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 13:12:06.768880217 (573h43m6.639963252s) [upload=502975]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:12:06.768884906 (573h43m6.639967941s) [upload=502975]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:12:07.067188330 (573h43m6.938271365s) [upload=502975]: Done uploading segments...
Dec 29 13:12:07.067251099 (573h43m6.938334134s) [upload=502975]: Upload finished. err: <nil>
Duration: 15.361017105s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
51 out of 60 is failed (threshold=6) [XXXXXX.XXXXXXXXXXXXXXX.XXXXXXX.X.XX.XXX.XX.XXXXXXX.XX.XXXXXX] duration:
5.001675619s trace:
https://jaeger.dev.storj.io/trace/39db0db6cea80c64
FAIL (2025-12-29 13:13:07 UTC)
orbiter warning: context deadline exceeded
Duration: 5.0017021s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.......X......................X.....X.....................XX] duration:
1m0.002105108s trace:
https://jaeger.dev.storj.io/trace/77d26141ea11d228
FAIL (2025-12-29 13:13:02 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12J2weU7Bmsr3RdQQ9QMuadT96XReBkyxTeaLJ2SEqVGGFX1PWu, Piece ID: JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ) context deadline exceeded; piecestore: (Node ID: 125sm6iHV94mJFLWrKeyRMH4b1qSMo32nDuW8uQr71xUL7Cema2, Piece ID: INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA) context deadline exceeded; piecestore: (Node ID: 12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL, Piece ID: Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA) context deadline exceeded
======LOGS======
Q2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.692912334 (573h43m3.563995379s) [noise=true , address=77.90.31.16:4759 , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.693322595 (573h43m3.56440564s) [noise=true , address=77.90.31.15:4711 , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.693739649 (573h43m3.564822694s) [noise=true , address=45.91.116.6:4981 , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.694315887 (573h43m3.565398942s) [noise=true , address=145.79.227.4:4220 , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.694765222 (573h43m3.565848267s) [noise=true , address=145.79.227.29:4952 , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.695202545 (573h43m3.56628559s) [noise=true , address=145.79.227.40:4058 , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.695687528 (573h43m3.566770584s) [noise=true , address=77.90.31.19:4857 , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.696135221 (573h43m3.567218276s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.696703733 (573h43m3.567786778s) [noise=true , address=64.49.13.170:4720 , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.698418298 (573h43m3.569501343s) [noise=true , address=145.79.227.36:4520 , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.700198829 (573h43m3.571281874s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.701993145 (573h43m3.57307619s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.703884687 (573h43m3.574967732s) [noise=true , address=145.79.227.33:4893 , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.705687109 (573h43m3.576770154s) [noise=true , address=45.91.116.10:4094 , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:03.707557651 (573h43m3.578640696s) [noise=true , address=145.79.227.43:4164 , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Uploading piece...
Dec 29 13:12:05.168247719 (573h43m5.039330754s) [upload=502975]: Next returned nil segment
Dec 29 13:12:05.168259962 (573h43m5.039343007s) [upload=502975]: Waiting for error group managing segments...
Dec 29 13:12:05.558873127 (573h43m5.429956162s) [noise=true , address=145.79.227.4:4220 , piece_id=SHHJ7EW7DTNOZXVBP6A64VIX7P6AQN62TBCUT3KTH4MIB6UXY3BA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.563024312 (573h43m5.434107347s) [noise=true , address=64.49.13.170:4720 , piece_id=HF6X3OT2EPPDE3DWAPWABVZUR3ERDBWXHG3NGNKEWZAD7G5AN4TA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.564115719 (573h43m5.435198755s) [noise=true , address=145.79.227.36:4520 , piece_id=MYJSXUH563JFUW5Q7YUIWL7466G6ASY5WXR3T2PGUFFCTRTOBYJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.593873402 (573h43m5.464956447s) [noise=true , address=145.79.227.29:4952 , piece_id=DKYDB2R4DNCGH6T3SLRR2MOZZUPVZZVRL2BKGYOBYXJJVPSZEJ4Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.898194604 (573h43m5.769277639s) [noise=true , address=45.91.116.10:4094 , piece_id=6UUIZNTESBBWJMDQ5ZTCHU7S6YKM4GDKO4ZAC6T2FUTPVGFFDS3A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:05.912845292 (573h43m5.783928337s) [noise=true , address=77.90.31.19:4857 , piece_id=Q35LXGPTOAG3R2UL3JLL7DNHHD4PSMEULYW6G3SPXKNVFZ5VTNRA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.426532207 (573h43m6.297615252s) [noise=true , address=193.149.164.143:4699 , piece_id=UDFDCYO25MOJBTMDE6F2WILXE3HLOB75UJZ4LW7ZMNSCOBJI6UJA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.449589562 (573h43m6.320672607s) [noise=true , address=77.90.31.15:4711 , piece_id=JITX25FNZ4JIKYXQY345UX3SQUCPDWJ25NJQZ2TXGFRLVCRYXUOQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.453186841 (573h43m6.324269876s) [noise=true , address=145.79.227.40:4058 , piece_id=KRHH2QZKSDPWTSSKYDVDWGKFJD3LPGP6U7KI2PPJ5T3TXYIUGJWA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.510769276 (573h43m6.381852321s) [noise=true , address=145.79.227.43:4164 , piece_id=J6Q4ZDOIIIBA2FXGZZTD6AYS7KGAWSRXXUBAPDPQWYKNU5GNXBBQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.518868879 (573h43m6.389951924s) [noise=true , address=145.79.227.33:4893 , piece_id=XGLX3DFNAY4F4GPGCGZ4ONDJ7NK62SAKX3PBRWHWI74EF3ZFMUSA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.519238043 (573h43m6.390321098s) [noise=true , address=45.91.116.6:4981 , piece_id=INTVU2EYTU3DIM6WQFQEGO6IEYH3YFP5U2VEN7A5KPTOKU2DWOOA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.523211328 (573h43m6.394294373s) [noise=true , address=77.90.31.16:4759 , piece_id=ARC4HRP57U5VQXZTT63QYLXBPZURI5VMJBIQZOFKGLJBQ3K3KGLQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.767869783 (573h43m6.638952818s) [noise=true , address=145.79.227.34:4924 , piece_id=MAQXQTAGXQ7AEPUCESUSUGMHSDMMXS7HHKAEA7B3HW5X66PDUF5A, seg_pos={0 0}, upload=502975]: Done uploading piece. err: <nil>
Dec 29 13:12:06.767889431 (573h43m6.638972476s) [seg_pos={0 0}, upload=502975]: Segment reached optimal threshold of 14 pieces.
Dec 29 13:12:06.768102116 (573h43m6.639185161s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1preT33ei4yVXXYVLHhaUbHbiyL14oHFEiof6cCWPig83W6CT3): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:06.768108257 (573h43m6.639191302s) [noise=true , address=45.91.116.8:4051 , piece_id=GDJFLQCHUF7BFQ7BVSNTVJ3TYZHYVOO6YYMO3KUIUSIEPAF4SFDQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768219208 (573h43m6.639302254s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:06.768224789 (573h43m6.639307824s) [noise=true , address=77.90.31.18:4821 , piece_id=LO5W47FHYOVGY5Q6BH5L6KX77HES3XRAFUTPI45HYPJZDO3YHLCQ, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768384213 (573h43m6.639467258s) [noise=true , address=64.49.13.169:4711 , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48802->64.49.13.169:4711: use of closed network connection
Dec 29 13:12:06.768399261 (573h43m6.639482296s) [noise=true , address=64.49.13.169:4711 , piece_id=4F4T4YKXMT7NF2C5KBUGJHQJGZL4VCKNKYB266PCVI3UACHJXZ5Q, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768732636 (573h43m6.639815671s) [noise=true , address=64.49.13.169:4709 , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:35310->64.49.13.169:4709: use of closed network connection
Dec 29 13:12:06.768745460 (573h43m6.639828505s) [noise=true , address=64.49.13.169:4709 , piece_id=TBHHX5ES2A6NQRGP25HIC7J7IBWLOJGQ2LO7MUXLHE5YET7BHLLA, seg_pos={0 0}, upload=502975]: Long tail cancellation due to optimal threshold
Dec 29 13:12:06.768784586 (573h43m6.639867631s) [seg_pos={0 0}, upload=502975]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 13:12:06.768880217 (573h43m6.639963252s) [upload=502975]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:12:06.768884906 (573h43m6.639967941s) [upload=502975]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:12:07.067188330 (573h43m6.938271365s) [upload=502975]: Done uploading segments...
Dec 29 13:12:07.067251099 (573h43m6.938334134s) [upload=502975]: Upload finished. err: <nil>
Duration: 1m0.00213246s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
30 out of 60 is failed (threshold=3) [..X....XXXX..XX..X...XX.X.XX...X.X.X.XXX.X.X.X.XX..X.XXX.XX.] duration:
6.034578821s trace:
https://jaeger.dev.storj.io/trace/4d0a30f64135bb3
FAIL (2025-12-29 13:12:57 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 29 13:12:04.985229816 (573h43m4.856312861s) [upload=502976]: Starting upload
Dec 29 13:12:04.985377688 (573h43m4.856460743s) [upload=502976]: Done starting upload
Dec 29 13:12:04.985451167 (573h43m4.856534243s) [upload=502976]: Uploading segments...
Dec 29 13:12:04.985745158 (573h43m4.856828193s) [upload=502976]: Got next segment. Inline: false
Dec 29 13:12:04.985790995 (573h43m4.85687404s) [upload=502976]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 13:12:04.985825440 (573h43m4.856908495s) [upload=502976]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 13:12:05.233743983 (573h43m5.104827028s) [seg_pos={0 0}, upload=502976]: Begin upload segment...
Dec 29 13:12:05.233879912 (573h43m5.104962957s) [noise=true , address=145.79.227.10:4401 , piece_id=DVRBJ3ZFYNR2GNTSM7ZBPZZKGSDDWR4XCFJ72VRBXBE2ZDMSFYYQ, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:05.233976326 (573h43m5.105059381s) [seg_pos={0 0}, upload=502976]: Done begin upload segment.
Dec 29 13:12:05.234045397 (573h43m5.105128432s) [noise=true , address=145.79.227.37:4566 , piece_id=ISMGHRTBVSAHHRKNV2PJV7WDAEKYPT6YLXAV4VBL7VEAEHCLUABQ, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:05.234761752 (573h43m5.105844787s) [noise=true , address=77.90.31.19:4844 , piece_id=CX7T4OCPGQIPVZ7PQG7YAC56DG5XPIGPBVOJUOV7KD77N6ZHNS4A, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:05.235202040 (573h43m5.106285075s) [noise=true , address=77.90.31.17:4773 , piece_id=MU5NJS6HG5X2ZZLUJJROZEWIU2RPDTZMHCTU6HE2QW3VGRORPZWQ, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:05.235697533 (573h43m5.106780578s) [noise=true , address=145.79.227.38:4979 , piece_id=WV7YS3MGMD2KYAA3ORTOS6YGK3KQZTUG4XKFNQXVOOH4THF5WCRA, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:05.236310140 (573h43m5.107393185s) [noise=true , address=145.79.227.20:4437 , piece_id=AX7WIS6EWJWZBPQ7JAPCUNDVZEL64FDM5GOVEK6V4QPKFORYO6CA, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:05.236905073 (573h43m5.107988108s) [noise=true , address=45.91.116.6:4976 , piece_id=ONS6LPVAFX2BKHENMZAZZID3WZOPQV7CJ2NU3VG2W2LHB2SVISNA, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:05.244302288 (573h43m5.115385333s) [noise=true , address=145.79.227.11:4732 , piece_id=FUDW7B6RL3MSTOTUAOFNGFV4OSPZUIVW574H5KP7W7CUAGXKZP4Q, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:05.245288366 (573h43m5.116371401s) [noise=true , address=145.79.227.51:4410 , piece_id=3K57LINT2MCQEUYY3MQWW7ZMW7IHPTQLESIKW4KHJJYYXDKDOPZQ, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:05.247332598 (573h43m5.118415633s) [noise=true , address=145.79.227.15:4876 , piece_id=P62GW63T7PUKQJ2M5424CUMOXKMRH4J366Z5ULETXLRKDVT2D7FQ, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:05.248844728 (573h43m5.119927773s) [noise=true , address=145.79.227.19:4079 , piece_id=NUJALKQUNVJSHQUX5JPAJX3HGFFEXTYNVOH2PUKGF73ZFKDDQFHQ, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:05.249290426 (573h43m5.120373471s) [noise=true , address=145.79.227.35:4481 , piece_id=TDHAUE3DN3DCBXXI66NFKCXU63ZCAQJSZAYU56IMSGX5BQJUJFZQ, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:05.250997606 (573h43m5.122080641s) [noise=true , address=77.90.31.14:4661 , piece_id=4GB66L3GH5YEAZMYITBDPKQT3I6GKJ2KQXEZ77PUM3F3TRZ3RDYQ, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:05.251918220 (573h43m5.123001255s) [noise=true , address=145.79.227.18:4007 , piece_id=DY5QZOE7ZNROSR6SPCHUITETGKCZZ4ZGMIKHKULKU7JGIUXZ76DQ, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:05.253074702 (573h43m5.124157737s) [noise=true , address=45.91.116.10:4082 , piece_id=JRDRU2EYT2TFZFXVQG6RBGTQ4PADBQDGSJOQBFVCKXRIHH5SAT2Q, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:05.254594075 (573h43m5.12567711s) [noise=true , address=145.79.227.7:4293 , piece_id=FSJYLIH2RXZY5XJUTQTMUXR6PLGXTPXHSADBBRYTWCWK6DAIXJBA, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:06.939401444 (573h43m6.81048448s) [upload=502976]: Next returned nil segment
Dec 29 13:12:06.939422725 (573h43m6.81050576s) [upload=502976]: Waiting for error group managing segments...
Dec 29 13:12:07.219898942 (573h43m7.090981987s) [noise=true , address=145.79.227.18:4007 , piece_id=DY5QZOE7ZNROSR6SPCHUITETGKCZZ4ZGMIKHKULKU7JGIUXZ76DQ, seg_pos={0 0}, upload=502976]: Done uploading piece. err: <nil>
Dec 29 13:12:07.230318393 (573h43m7.101401429s) [noise=true , address=145.79.227.20:4437 , piece_id=AX7WIS6EWJWZBPQ7JAPCUNDVZEL64FDM5GOVEK6V4QPKFORYO6CA, seg_pos={0 0}, upload=502976]: Done uploading piece. err: <nil>
Dec 29 13:12:07.230346387 (573h43m7.101429422s) [noise=true , address=145.79.227.10:4401 , piece_id=DVRBJ3ZFYNR2GNTSM7ZBPZZKGSDDWR4XCFJ72VRBXBE2ZDMSFYYQ, seg_pos={0 0}, upload=502976]: Done uploading piece. err: <nil>
Dec 29 13:12:07.235223112 (573h43m7.106306157s) [noise=true , address=145.79.227.15:4876 , piece_id=P62GW63T7PUKQJ2M5424CUMOXKMRH4J366Z5ULETXLRKDVT2D7FQ, seg_pos={0 0}, upload=502976]: Done uploading piece. err: <nil>
Dec 29 13:12:07.235504668 (573h43m7.106587703s) [noise=true , address=145.79.227.38:4979 , piece_id=WV7YS3MGMD2KYAA3ORTOS6YGK3KQZTUG4XKFNQXVOOH4THF5WCRA, seg_pos={0 0}, upload=502976]: Done uploading piece. err: <nil>
Dec 29 13:12:07.236420412 (573h43m7.107503448s) [noise=true , address=145.79.227.51:4410 , piece_id=3K57LINT2MCQEUYY3MQWW7ZMW7IHPTQLESIKW4KHJJYYXDKDOPZQ, seg_pos={0 0}, upload=502976]: Done uploading piece. err: <nil>
Dec 29 13:12:07.252096952 (573h43m7.123180007s) [noise=true , address=145.79.227.37:4566 , piece_id=ISMGHRTBVSAHHRKNV2PJV7WDAEKYPT6YLXAV4VBL7VEAEHCLUABQ, seg_pos={0 0}, upload=502976]: Done uploading piece. err: <nil>
Dec 29 13:12:07.252314717 (573h43m7.123397762s) [noise=true , address=145.79.227.11:4732 , piece_id=FUDW7B6RL3MSTOTUAOFNGFV4OSPZUIVW574H5KP7W7CUAGXKZP4Q, seg_pos={0 0}, upload=502976]: Done uploading piece. err: <nil>
Dec 29 13:12:07.258573716 (573h43m7.129656751s) [noise=true , address=145.79.227.19:4079 , piece_id=NUJALKQUNVJSHQUX5JPAJX3HGFFEXTYNVOH2PUKGF73ZFKDDQFHQ, seg_pos={0 0}, upload=502976]: Done uploading piece. err: <nil>
Dec 29 13:12:07.265694585 (573h43m7.13677762s) [noise=true , address=145.79.227.7:4293 , piece_id=FSJYLIH2RXZY5XJUTQTMUXR6PLGXTPXHSADBBRYTWCWK6DAIXJBA, seg_pos={0 0}, upload=502976]: Done uploading piece. err: <nil>
Dec 29 13:12:07.882198598 (573h43m7.753281643s) [noise=true , address=45.91.116.10:4082 , piece_id=JRDRU2EYT2TFZFXVQG6RBGTQ4PADBQDGSJOQBFVCKXRIHH5SAT2Q, seg_pos={0 0}, upload=502976]: Done uploading piece. err: <nil>
Dec 29 13:12:08.193700432 (573h43m8.064783467s) [noise=true , address=77.90.31.17:4773 , piece_id=MU5NJS6HG5X2ZZLUJJROZEWIU2RPDTZMHCTU6HE2QW3VGRORPZWQ, seg_pos={0 0}, upload=502976]: Done uploading piece. err: <nil>
Dec 29 13:12:08.385481111 (573h43m8.256564146s) [noise=true , address=45.91.116.6:4976 , piece_id=ONS6LPVAFX2BKHENMZAZZID3WZOPQV7CJ2NU3VG2W2LHB2SVISNA, seg_pos={0 0}, upload=502976]: Done uploading piece. err: <nil>
Dec 29 13:12:08.385506859 (573h43m8.256589894s) [seg_pos={0 0}, upload=502976]: Segment reached optimal threshold of 13 pieces.
Dec 29 13:12:08.385722060 (573h43m8.256805095s) [noise=true , address=77.90.31.14:4661 , piece_id=4GB66L3GH5YEAZMYITBDPKQT3I6GKJ2KQXEZ77PUM3F3TRZ3RDYQ, seg_pos={0 0}, upload=502976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:08.385728011 (573h43m8.256811046s) [noise=true , address=77.90.31.14:4661 , piece_id=4GB66L3GH5YEAZMYITBDPKQT3I6GKJ2KQXEZ77PUM3F3TRZ3RDYQ, seg_pos={0 0}, upload=502976]: Long tail cancellation due to optimal threshold
Dec 29 13:12:08.385829484 (573h43m8.256912529s) [noise=true , address=77.90.31.19:4844 , piece_id=CX7T4OCPGQIPVZ7PQG7YAC56DG5XPIGPBVOJUOV7KD77N6ZHNS4A, seg_pos={0 0}, upload=502976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:08.385845464 (573h43m8.256928499s) [noise=true , address=77.90.31.19:4844 , piece_id=CX7T4OCPGQIPVZ7PQG7YAC56DG5XPIGPBVOJUOV7KD77N6ZHNS4A, seg_pos={0 0}, upload=502976]: Long tail cancellation due to optimal threshold
Dec 29 13:12:08.386004277 (573h43m8.257087312s) [noise=true , address=145.79.227.35:4481 , piece_id=TDHAUE3DN3DCBXXI66NFKCXU63ZCAQJSZAYU56IMSGX5BQJUJFZQ, seg_pos={0 0}, upload=502976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:08.386011150 (573h43m8.257094185s) [noise=true , address=145.79.227.35:4481 , piece_id=TDHAUE3DN3DCBXXI66NFKCXU63ZCAQJSZAYU56IMSGX5BQJUJFZQ, seg_pos={0 0}, upload=502976]: Long tail cancellation due to optimal threshold
Dec 29 13:12:08.386028823 (573h43m8.257111858s) [seg_pos={0 0}, upload=502976]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 13:12:08.386168599 (573h43m8.257251644s) [upload=502976]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:12:08.386176204 (573h43m8.257259249s) [upload=502976]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:12:08.691718291 (573h43m8.562801346s) [upload=502976]: Done uploading segments...
Dec 29 13:12:08.691772264 (573h43m8.562855299s) [upload=502976]: Upload finished. err: <nil>
Duration: 6.034689321s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.......................X......................X....X.X....X.] duration:
21.977194961s trace:
https://jaeger.dev.storj.io/trace/704f4992ad0b112c
FAIL (2025-12-29 13:12:56 UTC)
orbiter warning: context deadline exceeded
Duration: 21.97720998s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
51 out of 60 is failed (threshold=6) [XXXXXXX.XXXXXXXXXXXXXXX.XXXXXXX.X.XX.XXX.XX.XXXXXXX.XX.XXXXX] duration:
5.000827845s trace:
https://jaeger.dev.storj.io/trace/6da0f0f24a0f1eee
FAIL (2025-12-29 13:12:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000847382s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
31 out of 60 is failed (threshold=3) [X..X....XXXX..XX..X...XX.X.XX...X.X.X.XXX.X.X.X.XX..X.XXX.XX] duration:
30.003551543s trace:
https://jaeger.dev.storj.io/trace/4d0a30f64135bb3
FAIL (2025-12-29 13:12:34 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 29 13:12:04.985229816 (573h43m4.856312861s) [upload=502976]: Starting upload
Dec 29 13:12:04.985377688 (573h43m4.856460743s) [upload=502976]: Done starting upload
Dec 29 13:12:04.985451167 (573h43m4.856534243s) [upload=502976]: Uploading segments...
Dec 29 13:12:04.985745158 (573h43m4.856828193s) [upload=502976]: Got next segment. Inline: false
Dec 29 13:12:04.985790995 (573h43m4.85687404s) [upload=502976]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 13:12:04.985825440 (573h43m4.856908495s) [upload=502976]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 13:12:05.233743983 (573h43m5.104827028s) [seg_pos={0 0}, upload=502976]: Begin upload segment...
Dec 29 13:12:05.233879912 (573h43m5.104962957s) [noise=true , address=145.79.227.10:4401 , piece_id=DVRBJ3ZFYNR2GNTSM7ZBPZZKGSDDWR4XCFJ72VRBXBE2ZDMSFYYQ, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:05.233976326 (573h43m5.105059381s) [seg_pos={0 0}, upload=502976]: Done begin upload segment.
Dec 29 13:12:05.234045397 (573h43m5.105128432s) [noise=true , address=145.79.227.37:4566 , piece_id=ISMGHRTBVSAHHRKNV2PJV7WDAEKYPT6YLXAV4VBL7VEAEHCLUABQ, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:05.234761752 (573h43m5.105844787s) [noise=true , address=77.90.31.19:4844 , piece_id=CX7T4OCPGQIPVZ7PQG7YAC56DG5XPIGPBVOJUOV7KD77N6ZHNS4A, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:05.235202040 (573h43m5.106285075s) [noise=true , address=77.90.31.17:4773 , piece_id=MU5NJS6HG5X2ZZLUJJROZEWIU2RPDTZMHCTU6HE2QW3VGRORPZWQ, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:05.235697533 (573h43m5.106780578s) [noise=true , address=145.79.227.38:4979 , piece_id=WV7YS3MGMD2KYAA3ORTOS6YGK3KQZTUG4XKFNQXVOOH4THF5WCRA, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:05.236310140 (573h43m5.107393185s) [noise=true , address=145.79.227.20:4437 , piece_id=AX7WIS6EWJWZBPQ7JAPCUNDVZEL64FDM5GOVEK6V4QPKFORYO6CA, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:05.236905073 (573h43m5.107988108s) [noise=true , address=45.91.116.6:4976 , piece_id=ONS6LPVAFX2BKHENMZAZZID3WZOPQV7CJ2NU3VG2W2LHB2SVISNA, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:05.244302288 (573h43m5.115385333s) [noise=true , address=145.79.227.11:4732 , piece_id=FUDW7B6RL3MSTOTUAOFNGFV4OSPZUIVW574H5KP7W7CUAGXKZP4Q, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:05.245288366 (573h43m5.116371401s) [noise=true , address=145.79.227.51:4410 , piece_id=3K57LINT2MCQEUYY3MQWW7ZMW7IHPTQLESIKW4KHJJYYXDKDOPZQ, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:05.247332598 (573h43m5.118415633s) [noise=true , address=145.79.227.15:4876 , piece_id=P62GW63T7PUKQJ2M5424CUMOXKMRH4J366Z5ULETXLRKDVT2D7FQ, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:05.248844728 (573h43m5.119927773s) [noise=true , address=145.79.227.19:4079 , piece_id=NUJALKQUNVJSHQUX5JPAJX3HGFFEXTYNVOH2PUKGF73ZFKDDQFHQ, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:05.249290426 (573h43m5.120373471s) [noise=true , address=145.79.227.35:4481 , piece_id=TDHAUE3DN3DCBXXI66NFKCXU63ZCAQJSZAYU56IMSGX5BQJUJFZQ, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:05.250997606 (573h43m5.122080641s) [noise=true , address=77.90.31.14:4661 , piece_id=4GB66L3GH5YEAZMYITBDPKQT3I6GKJ2KQXEZ77PUM3F3TRZ3RDYQ, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:05.251918220 (573h43m5.123001255s) [noise=true , address=145.79.227.18:4007 , piece_id=DY5QZOE7ZNROSR6SPCHUITETGKCZZ4ZGMIKHKULKU7JGIUXZ76DQ, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:05.253074702 (573h43m5.124157737s) [noise=true , address=45.91.116.10:4082 , piece_id=JRDRU2EYT2TFZFXVQG6RBGTQ4PADBQDGSJOQBFVCKXRIHH5SAT2Q, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:05.254594075 (573h43m5.12567711s) [noise=true , address=145.79.227.7:4293 , piece_id=FSJYLIH2RXZY5XJUTQTMUXR6PLGXTPXHSADBBRYTWCWK6DAIXJBA, seg_pos={0 0}, upload=502976]: Uploading piece...
Dec 29 13:12:06.939401444 (573h43m6.81048448s) [upload=502976]: Next returned nil segment
Dec 29 13:12:06.939422725 (573h43m6.81050576s) [upload=502976]: Waiting for error group managing segments...
Dec 29 13:12:07.219898942 (573h43m7.090981987s) [noise=true , address=145.79.227.18:4007 , piece_id=DY5QZOE7ZNROSR6SPCHUITETGKCZZ4ZGMIKHKULKU7JGIUXZ76DQ, seg_pos={0 0}, upload=502976]: Done uploading piece. err: <nil>
Dec 29 13:12:07.230318393 (573h43m7.101401429s) [noise=true , address=145.79.227.20:4437 , piece_id=AX7WIS6EWJWZBPQ7JAPCUNDVZEL64FDM5GOVEK6V4QPKFORYO6CA, seg_pos={0 0}, upload=502976]: Done uploading piece. err: <nil>
Dec 29 13:12:07.230346387 (573h43m7.101429422s) [noise=true , address=145.79.227.10:4401 , piece_id=DVRBJ3ZFYNR2GNTSM7ZBPZZKGSDDWR4XCFJ72VRBXBE2ZDMSFYYQ, seg_pos={0 0}, upload=502976]: Done uploading piece. err: <nil>
Dec 29 13:12:07.235223112 (573h43m7.106306157s) [noise=true , address=145.79.227.15:4876 , piece_id=P62GW63T7PUKQJ2M5424CUMOXKMRH4J366Z5ULETXLRKDVT2D7FQ, seg_pos={0 0}, upload=502976]: Done uploading piece. err: <nil>
Dec 29 13:12:07.235504668 (573h43m7.106587703s) [noise=true , address=145.79.227.38:4979 , piece_id=WV7YS3MGMD2KYAA3ORTOS6YGK3KQZTUG4XKFNQXVOOH4THF5WCRA, seg_pos={0 0}, upload=502976]: Done uploading piece. err: <nil>
Dec 29 13:12:07.236420412 (573h43m7.107503448s) [noise=true , address=145.79.227.51:4410 , piece_id=3K57LINT2MCQEUYY3MQWW7ZMW7IHPTQLESIKW4KHJJYYXDKDOPZQ, seg_pos={0 0}, upload=502976]: Done uploading piece. err: <nil>
Dec 29 13:12:07.252096952 (573h43m7.123180007s) [noise=true , address=145.79.227.37:4566 , piece_id=ISMGHRTBVSAHHRKNV2PJV7WDAEKYPT6YLXAV4VBL7VEAEHCLUABQ, seg_pos={0 0}, upload=502976]: Done uploading piece. err: <nil>
Dec 29 13:12:07.252314717 (573h43m7.123397762s) [noise=true , address=145.79.227.11:4732 , piece_id=FUDW7B6RL3MSTOTUAOFNGFV4OSPZUIVW574H5KP7W7CUAGXKZP4Q, seg_pos={0 0}, upload=502976]: Done uploading piece. err: <nil>
Dec 29 13:12:07.258573716 (573h43m7.129656751s) [noise=true , address=145.79.227.19:4079 , piece_id=NUJALKQUNVJSHQUX5JPAJX3HGFFEXTYNVOH2PUKGF73ZFKDDQFHQ, seg_pos={0 0}, upload=502976]: Done uploading piece. err: <nil>
Dec 29 13:12:07.265694585 (573h43m7.13677762s) [noise=true , address=145.79.227.7:4293 , piece_id=FSJYLIH2RXZY5XJUTQTMUXR6PLGXTPXHSADBBRYTWCWK6DAIXJBA, seg_pos={0 0}, upload=502976]: Done uploading piece. err: <nil>
Dec 29 13:12:07.882198598 (573h43m7.753281643s) [noise=true , address=45.91.116.10:4082 , piece_id=JRDRU2EYT2TFZFXVQG6RBGTQ4PADBQDGSJOQBFVCKXRIHH5SAT2Q, seg_pos={0 0}, upload=502976]: Done uploading piece. err: <nil>
Dec 29 13:12:08.193700432 (573h43m8.064783467s) [noise=true , address=77.90.31.17:4773 , piece_id=MU5NJS6HG5X2ZZLUJJROZEWIU2RPDTZMHCTU6HE2QW3VGRORPZWQ, seg_pos={0 0}, upload=502976]: Done uploading piece. err: <nil>
Dec 29 13:12:08.385481111 (573h43m8.256564146s) [noise=true , address=45.91.116.6:4976 , piece_id=ONS6LPVAFX2BKHENMZAZZID3WZOPQV7CJ2NU3VG2W2LHB2SVISNA, seg_pos={0 0}, upload=502976]: Done uploading piece. err: <nil>
Dec 29 13:12:08.385506859 (573h43m8.256589894s) [seg_pos={0 0}, upload=502976]: Segment reached optimal threshold of 13 pieces.
Dec 29 13:12:08.385722060 (573h43m8.256805095s) [noise=true , address=77.90.31.14:4661 , piece_id=4GB66L3GH5YEAZMYITBDPKQT3I6GKJ2KQXEZ77PUM3F3TRZ3RDYQ, seg_pos={0 0}, upload=502976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:08.385728011 (573h43m8.256811046s) [noise=true , address=77.90.31.14:4661 , piece_id=4GB66L3GH5YEAZMYITBDPKQT3I6GKJ2KQXEZ77PUM3F3TRZ3RDYQ, seg_pos={0 0}, upload=502976]: Long tail cancellation due to optimal threshold
Dec 29 13:12:08.385829484 (573h43m8.256912529s) [noise=true , address=77.90.31.19:4844 , piece_id=CX7T4OCPGQIPVZ7PQG7YAC56DG5XPIGPBVOJUOV7KD77N6ZHNS4A, seg_pos={0 0}, upload=502976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:08.385845464 (573h43m8.256928499s) [noise=true , address=77.90.31.19:4844 , piece_id=CX7T4OCPGQIPVZ7PQG7YAC56DG5XPIGPBVOJUOV7KD77N6ZHNS4A, seg_pos={0 0}, upload=502976]: Long tail cancellation due to optimal threshold
Dec 29 13:12:08.386004277 (573h43m8.257087312s) [noise=true , address=145.79.227.35:4481 , piece_id=TDHAUE3DN3DCBXXI66NFKCXU63ZCAQJSZAYU56IMSGX5BQJUJFZQ, seg_pos={0 0}, upload=502976]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF
Dec 29 13:12:08.386011150 (573h43m8.257094185s) [noise=true , address=145.79.227.35:4481 , piece_id=TDHAUE3DN3DCBXXI66NFKCXU63ZCAQJSZAYU56IMSGX5BQJUJFZQ, seg_pos={0 0}, upload=502976]: Long tail cancellation due to optimal threshold
Dec 29 13:12:08.386028823 (573h43m8.257111858s) [seg_pos={0 0}, upload=502976]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 13:12:08.386168599 (573h43m8.257251644s) [upload=502976]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:12:08.386176204 (573h43m8.257259249s) [upload=502976]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:12:08.691718291 (573h43m8.562801346s) [upload=502976]: Done uploading segments...
Dec 29 13:12:08.691772264 (573h43m8.562855299s) [upload=502976]: Upload finished. err: <nil>
Duration: 30.003572973s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [........................X......................X....X.X....X] duration:
30.001106435s trace:
https://jaeger.dev.storj.io/trace/704f4992ad0b112c
FAIL (2025-12-29 13:12:05 UTC)
orbiter warning: context deadline exceeded
Duration: 30.00112981s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
30 out of 60 is failed (threshold=3) [.X..X....XXXX..XX..X...XX.X.XX...X.X.X.XXX.X.X.X.XX..X.XXX.X] duration:
30.003049273s trace:
https://jaeger.dev.storj.io/trace/6a0a1e345b171f9c
FAIL (2025-12-29 13:12:02 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: ABMPT24QKPWNM2SBAKYYM45GWHAUV5XZ5ZNLPIQRYE3OJJPU43PQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: DLKVBXU4FOUXAU5NPQ2KRRUP66ECHHC54APC4WCCVSUZEZ3A2HSA) context deadline exceeded
======LOGS======
g segments...
Dec 29 13:11:32.800784445 (573h42m32.67186748s) [upload=502968]: Got next segment. Inline: false
Dec 29 13:11:32.800813881 (573h42m32.671896916s) [upload=502968]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 13:11:32.800819842 (573h42m32.671902877s) [upload=502968]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 13:11:33.046210672 (573h42m32.917293707s) [seg_pos={0 0}, upload=502968]: Begin upload segment...
Dec 29 13:11:33.046260046 (573h42m32.917343081s) [seg_pos={0 0}, upload=502968]: Done begin upload segment.
Dec 29 13:11:33.046288891 (573h42m32.917371926s) [noise=true , address=145.79.227.6:4258 , piece_id=MTERNZN4GQQICZUKD37M5MYKNVJFEBZGGTAECCGWPM7PI6AZO3LA, seg_pos={0 0}, upload=502968]: Uploading piece...
Dec 29 13:11:33.046300994 (573h42m32.917384039s) [noise=true , address=145.79.227.3:4150 , piece_id=JISVRPZLMBSQNVOBZAGILDKGPJMQUGQ6UHFLDDE4LYQOR34TNRDA, seg_pos={0 0}, upload=502968]: Uploading piece...
Dec 29 13:11:33.046387669 (573h42m32.917470714s) [noise=true , address=145.79.227.32:4834 , piece_id=XT2YHOLOE3MELOS7JWXWFZD4NBFBP32UASY7DJ3YZDNHL737DVOQ, seg_pos={0 0}, upload=502968]: Uploading piece...
Dec 29 13:11:33.047466063 (573h42m32.918549098s) [noise=true , address=145.79.227.47:4267 , piece_id=L4RH3JYWBIN3J5JPLTWTWC5C5O3DEQIKNIYHVHNXLJLNRGONL47A, seg_pos={0 0}, upload=502968]: Uploading piece...
Dec 29 13:11:33.048340558 (573h42m32.919423593s) [noise=true , address=145.79.227.45:4229 , piece_id=TBRPPDSR7P57SYXMKUHOWKL7B2OWHZDRDEIOO46D6PLHW57KOOGQ, seg_pos={0 0}, upload=502968]: Uploading piece...
Dec 29 13:11:33.048668391 (573h42m32.919751437s) [noise=true , address=45.91.116.9:4263 , piece_id=ABMPT24QKPWNM2SBAKYYM45GWHAUV5XZ5ZNLPIQRYE3OJJPU43PQ, seg_pos={0 0}, upload=502968]: Uploading piece...
Dec 29 13:11:33.048948164 (573h42m32.920031199s) [noise=true , address=145.79.227.2:4113 , piece_id=MVI7YFJJ22N2MTQBFYOJC35DHLVEDCFYRSZL4F7RWKBOILZBAMRA, seg_pos={0 0}, upload=502968]: Uploading piece...
Dec 29 13:11:33.049785039 (573h42m32.920868074s) [noise=true , address=145.79.227.46:4602 , piece_id=UTCGLTD63FCFLPAFUOKWYHA53IVP4QE2SPAZWNMRDSDURUPTSCDA, seg_pos={0 0}, upload=502968]: Uploading piece...
Dec 29 13:11:33.050110508 (573h42m32.921193543s) [noise=true , address=45.91.116.6:4975 , piece_id=DLKVBXU4FOUXAU5NPQ2KRRUP66ECHHC54APC4WCCVSUZEZ3A2HSA, seg_pos={0 0}, upload=502968]: Uploading piece...
Dec 29 13:11:33.051271228 (573h42m32.922354263s) [noise=true , address=145.79.227.37:4565 , piece_id=WQXH5ABTSCXZTB6ITBPG4QGCJXY7CXDW34APZRTXSSGD6DSLY5JQ, seg_pos={0 0}, upload=502968]: Uploading piece...
Dec 29 13:11:33.051316946 (573h42m32.922399981s) [noise=true , address=77.90.31.21:4877 , piece_id=T33XP4B7WQSNR3XGDWRZIXC6VT32CBIU2O5HF7LWR2Z73YO3HCAA, seg_pos={0 0}, upload=502968]: Uploading piece...
Dec 29 13:11:33.052454581 (573h42m32.923537646s) [noise=true , address=145.79.227.38:4977 , piece_id=KPDWDPF4DQAFDHXDSBXKRJDM65R3KNTG33BK4UDDAQWI4UKV5PTA, seg_pos={0 0}, upload=502968]: Uploading piece...
Dec 29 13:11:33.054471742 (573h42m32.925554777s) [noise=true , address=145.79.227.12:4767 , piece_id=FKO54DQZ3KDR4MWBR4QSEHWKFBO36PH5KIONZDFBFDN6MQ34BRUA, seg_pos={0 0}, upload=502968]: Uploading piece...
Dec 29 13:11:33.056104692 (573h42m32.927187727s) [noise=true , address=145.79.227.15:4876 , piece_id=PFUXLQ2WVHXN7O27MM27IRCV4QGNFKELNI5D36CN2IAMGUQ6M55Q, seg_pos={0 0}, upload=502968]: Uploading piece...
Dec 29 13:11:33.057645886 (573h42m32.928728921s) [noise=true , address=45.91.116.5:4297 , piece_id=TEVZIV5UOEQMMWZIW2ODEQSN2VNBVRP2UNCZIHWYSOTFHFIUB73A, seg_pos={0 0}, upload=502968]: Uploading piece...
Dec 29 13:11:33.058832185 (573h42m32.92991523s) [noise=true , address=145.79.227.14:4838 , piece_id=AWY4U6L7DVTVFZCZ5BUTDCOEAIHYDIGRXAQLCXMK5BB52X2DJHFA, seg_pos={0 0}, upload=502968]: Uploading piece...
Dec 29 13:11:34.757056573 (573h42m34.628139608s) [upload=502968]: Next returned nil segment
Dec 29 13:11:34.757117910 (573h42m34.628200955s) [upload=502968]: Waiting for error group managing segments...
Dec 29 13:11:35.030211439 (573h42m34.901294484s) [noise=true , address=145.79.227.3:4150 , piece_id=JISVRPZLMBSQNVOBZAGILDKGPJMQUGQ6UHFLDDE4LYQOR34TNRDA, seg_pos={0 0}, upload=502968]: Done uploading piece. err: <nil>
Dec 29 13:11:35.033691125 (573h42m34.90477416s) [noise=true , address=145.79.227.32:4834 , piece_id=XT2YHOLOE3MELOS7JWXWFZD4NBFBP32UASY7DJ3YZDNHL737DVOQ, seg_pos={0 0}, upload=502968]: Done uploading piece. err: <nil>
Dec 29 13:11:35.037214434 (573h42m34.908297479s) [noise=true , address=145.79.227.12:4767 , piece_id=FKO54DQZ3KDR4MWBR4QSEHWKFBO36PH5KIONZDFBFDN6MQ34BRUA, seg_pos={0 0}, upload=502968]: Done uploading piece. err: <nil>
Dec 29 13:11:35.041652315 (573h42m34.91273536s) [noise=true , address=145.79.227.46:4602 , piece_id=UTCGLTD63FCFLPAFUOKWYHA53IVP4QE2SPAZWNMRDSDURUPTSCDA, seg_pos={0 0}, upload=502968]: Done uploading piece. err: <nil>
Dec 29 13:11:35.042052056 (573h42m34.913135101s) [noise=true , address=145.79.227.14:4838 , piece_id=AWY4U6L7DVTVFZCZ5BUTDCOEAIHYDIGRXAQLCXMK5BB52X2DJHFA, seg_pos={0 0}, upload=502968]: Done uploading piece. err: <nil>
Dec 29 13:11:35.042899510 (573h42m34.913982555s) [noise=true , address=145.79.227.15:4876 , piece_id=PFUXLQ2WVHXN7O27MM27IRCV4QGNFKELNI5D36CN2IAMGUQ6M55Q, seg_pos={0 0}, upload=502968]: Done uploading piece. err: <nil>
Dec 29 13:11:35.043257471 (573h42m34.914340506s) [noise=true , address=145.79.227.47:4267 , piece_id=L4RH3JYWBIN3J5JPLTWTWC5C5O3DEQIKNIYHVHNXLJLNRGONL47A, seg_pos={0 0}, upload=502968]: Done uploading piece. err: <nil>
Dec 29 13:11:35.044194947 (573h42m34.915277982s) [noise=true , address=145.79.227.45:4229 , piece_id=TBRPPDSR7P57SYXMKUHOWKL7B2OWHZDRDEIOO46D6PLHW57KOOGQ, seg_pos={0 0}, upload=502968]: Done uploading piece. err: <nil>
Dec 29 13:11:35.047938655 (573h42m34.91902169s) [noise=true , address=145.79.227.2:4113 , piece_id=MVI7YFJJ22N2MTQBFYOJC35DHLVEDCFYRSZL4F7RWKBOILZBAMRA, seg_pos={0 0}, upload=502968]: Done uploading piece. err: <nil>
Dec 29 13:11:35.064863983 (573h42m34.935947018s) [noise=true , address=145.79.227.38:4977 , piece_id=KPDWDPF4DQAFDHXDSBXKRJDM65R3KNTG33BK4UDDAQWI4UKV5PTA, seg_pos={0 0}, upload=502968]: Done uploading piece. err: <nil>
Dec 29 13:11:35.074982690 (573h42m34.946065735s) [noise=true , address=145.79.227.6:4258 , piece_id=MTERNZN4GQQICZUKD37M5MYKNVJFEBZGGTAECCGWPM7PI6AZO3LA, seg_pos={0 0}, upload=502968]: Done uploading piece. err: <nil>
Dec 29 13:11:35.415729261 (573h42m35.286812296s) [noise=true , address=45.91.116.9:4263 , piece_id=ABMPT24QKPWNM2SBAKYYM45GWHAUV5XZ5ZNLPIQRYE3OJJPU43PQ, seg_pos={0 0}, upload=502968]: Done uploading piece. err: <nil>
Dec 29 13:11:35.460290383 (573h42m35.331373418s) [noise=true , address=45.91.116.6:4975 , piece_id=DLKVBXU4FOUXAU5NPQ2KRRUP66ECHHC54APC4WCCVSUZEZ3A2HSA, seg_pos={0 0}, upload=502968]: Done uploading piece. err: <nil>
Dec 29 13:11:35.460307796 (573h42m35.331390831s) [seg_pos={0 0}, upload=502968]: Segment reached optimal threshold of 13 pieces.
Dec 29 13:11:35.460459765 (573h42m35.33154281s) [noise=true , address=45.91.116.5:4297 , piece_id=TEVZIV5UOEQMMWZIW2ODEQSN2VNBVRP2UNCZIHWYSOTFHFIUB73A, seg_pos={0 0}, upload=502968]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: write tcp 188.166.225.170:56676->45.91.116.5:4297: use of closed network connection; write tcp 188.166.225.170:56676->45.91.116.5:4297: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56676->45.91.116.5:4297: use of closed network connection
Dec 29 13:11:35.460465677 (573h42m35.331548712s) [noise=true , address=45.91.116.5:4297 , piece_id=TEVZIV5UOEQMMWZIW2ODEQSN2VNBVRP2UNCZIHWYSOTFHFIUB73A, seg_pos={0 0}, upload=502968]: Long tail cancellation due to optimal threshold
Dec 29 13:11:35.460602888 (573h42m35.331685923s) [noise=true , address=145.79.227.37:4565 , piece_id=WQXH5ABTSCXZTB6ITBPG4QGCJXY7CXDW34APZRTXSSGD6DSLY5JQ, seg_pos={0 0}, upload=502968]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT): protocol: expected piece hash; context canceled; EOF
Dec 29 13:11:35.460635540 (573h42m35.331718575s) [noise=true , address=145.79.227.37:4565 , piece_id=WQXH5ABTSCXZTB6ITBPG4QGCJXY7CXDW34APZRTXSSGD6DSLY5JQ, seg_pos={0 0}, upload=502968]: Long tail cancellation due to optimal threshold
Dec 29 13:11:35.460654977 (573h42m35.331738012s) [noise=true , address=77.90.31.21:4877 , piece_id=T33XP4B7WQSNR3XGDWRZIXC6VT32CBIU2O5HF7LWR2Z73YO3HCAA, seg_pos={0 0}, upload=502968]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: write tcp 188.166.225.170:37486->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:37486->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37486->77.90.31.21:4877: use of closed network connection
Dec 29 13:11:35.460661860 (573h42m35.331744905s) [noise=true , address=77.90.31.21:4877 , piece_id=T33XP4B7WQSNR3XGDWRZIXC6VT32CBIU2O5HF7LWR2Z73YO3HCAA, seg_pos={0 0}, upload=502968]: Long tail cancellation due to optimal threshold
Dec 29 13:11:35.460676839 (573h42m35.331759874s) [seg_pos={0 0}, upload=502968]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 13:11:35.460733777 (573h42m35.331816812s) [upload=502968]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:11:35.460737854 (573h42m35.331820889s) [upload=502968]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:11:35.748397551 (573h42m35.619480586s) [upload=502968]: Done uploading segments...
Dec 29 13:11:35.748417279 (573h42m35.619500314s) [upload=502968]: Upload finished. err: <nil>
Duration: 30.003077757s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [........X......................X.....X.....................X] duration:
1m0.002179852s trace:
https://jaeger.dev.storj.io/trace/487a51248faea6d7
FAIL (2025-12-29 13:11:54 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx, Piece ID: OIW4FHJF7BESOLVLTUK5VOHSREK35T5M3SUKZHICW7XVN6CDW44Q) context deadline exceeded; piecestore: (Node ID: 12hE2JH2wY7TepnbCVpSxkTcVifqui3QK42tLh1neU2BxZ2BRFJ, Piece ID: JCF2JXHL3IYZEGYJ5YR37ZEKT27S4H7PDYNFP2PUKML7UVGTWZIQ) context deadline exceeded; piecestore: (Node ID: 1Wy976SVyCJwVBqfYneFHXjiRx8U2iKhdjzsbjyiAgsGUJS4nX, Piece ID: 2E6GESG6RHHBKPQNB2LBCMXQK2CR7N4DHGV4AQQUUMMCBBQVUGKQ) context deadline exceeded; piecestore: (Node ID: 12TxXc8GpnEATg9PMHcQxRFYUz7STpdzMuQsFXkiM9DgzchDRjJ, Piece ID: GTR3MK7AZIX3VFMZMJ5YERMNR5KTHI2LFE42MINU3HOU3RHMWK5A) context deadline exceeded
======LOGS======
seg_pos={0 0}, upload=502959]: Uploading piece...
Dec 29 13:10:55.175338811 (573h41m55.046421846s) [noise=true , address=45.91.116.10:4112 , piece_id=2E6GESG6RHHBKPQNB2LBCMXQK2CR7N4DHGV4AQQUUMMCBBQVUGKQ, seg_pos={0 0}, upload=502959]: Uploading piece...
Dec 29 13:10:55.175725317 (573h41m55.046808352s) [noise=true , address=64.49.13.169:4711 , piece_id=CTQNJBLYSYQU6SC4QOCF4EQGSAPRC7OHU2ON63V6A7TNPIT3PZZA, seg_pos={0 0}, upload=502959]: Uploading piece...
Dec 29 13:10:55.176273431 (573h41m55.047356476s) [noise=true , address=45.91.116.5:4327 , piece_id=VHAVSNFNQCBZAL77IF4Y665J5LT2PUIEGW7NZI26FGF6ZJCW24IA, seg_pos={0 0}, upload=502959]: Uploading piece...
Dec 29 13:10:55.177115154 (573h41m55.048198189s) [noise=true , address=45.91.116.13:4175 , piece_id=BHAA4Y5PONUKIYP7ZCUQX4ALUNOIB57GTEVSI5DXKJWAH3YRZAVA, seg_pos={0 0}, upload=502959]: Uploading piece...
Dec 29 13:10:55.177518180 (573h41m55.048601225s) [noise=true , address=193.149.164.143:4694 , piece_id=OIW4FHJF7BESOLVLTUK5VOHSREK35T5M3SUKZHICW7XVN6CDW44Q, seg_pos={0 0}, upload=502959]: Uploading piece...
Dec 29 13:10:55.178286644 (573h41m55.049369689s) [noise=true , address=145.79.227.40:4053 , piece_id=2KH5524YHDP6XVLSF7MQHJV5XLB2SXJDMUQALREV7XDNGP7T3A3Q, seg_pos={0 0}, upload=502959]: Uploading piece...
Dec 29 13:10:55.178848113 (573h41m55.049931158s) [noise=true , address=145.79.227.3:4184 , piece_id=5CSG4AMBAEH6K6NAERPU7FYYIGPTAD5BX6LAEH6Z35JODTUNXGUQ, seg_pos={0 0}, upload=502959]: Uploading piece...
Dec 29 13:10:55.179331453 (573h41m55.050414488s) [noise=true , address=145.79.227.49:4365 , piece_id=IBRPNRC5EUT5FPD55YWZ7E3VQWQZW3FIWQWYFW4WTEHKMAQKCLCQ, seg_pos={0 0}, upload=502959]: Uploading piece...
Dec 29 13:10:55.179990729 (573h41m55.051073774s) [noise=true , address=77.90.31.14:4691 , piece_id=GTR3MK7AZIX3VFMZMJ5YERMNR5KTHI2LFE42MINU3HOU3RHMWK5A, seg_pos={0 0}, upload=502959]: Uploading piece...
Dec 29 13:10:55.180435085 (573h41m55.05151812s) [noise=true , address=145.79.227.33:4872 , piece_id=375VDGMK3XNUHA5I6SXX66RSHWSBPRA2JLS3WGQNFGEWOHECLH3Q, seg_pos={0 0}, upload=502959]: Uploading piece...
Dec 29 13:10:55.180863901 (573h41m55.051946936s) [noise=true , address=64.49.13.140:4516 , piece_id=VQRJQW4MMFG3JVOKIA6IWBGTRJYW7WLTKPISOPZTYX5BUOCJMWLA, seg_pos={0 0}, upload=502959]: Uploading piece...
Dec 29 13:10:55.182158897 (573h41m55.053241932s) [noise=true , address=77.90.31.18:4821 , piece_id=RWDYWW3YSVKD3TARAI5BOBXC6RM7PBQNNR44JZSJKOHBTNTGAU4Q, seg_pos={0 0}, upload=502959]: Uploading piece...
Dec 29 13:10:55.182580140 (573h41m55.053663175s) [noise=true , address=145.79.227.30:4778 , piece_id=4RVR62UZNMHZX7TTUER5L5E6R2NED47CDKLWUUUROPTM7KTLEOFA, seg_pos={0 0}, upload=502959]: Uploading piece...
Dec 29 13:10:55.182891363 (573h41m55.053974398s) [noise=true , address=193.149.164.143:4699 , piece_id=DOBA6ISR7ICDWOERBGVKIKPW4F7ZEYWZKJPVD6QAEFVWIGBZK6ZQ, seg_pos={0 0}, upload=502959]: Uploading piece...
Dec 29 13:10:55.183206653 (573h41m55.054289688s) [noise=true , address=77.90.31.13:4637 , piece_id=ZT7OZ4HBHYFQ6RNS3IJOVFPJAM7SXNF4CJYDETCMUQET2OQ7ISAA, seg_pos={0 0}, upload=502959]: Uploading piece...
Dec 29 13:10:56.545119170 (573h41m56.416202205s) [upload=502959]: Next returned nil segment
Dec 29 13:10:56.545132777 (573h41m56.416215812s) [upload=502959]: Waiting for error group managing segments...
Dec 29 13:10:56.891041355 (573h41m56.762124411s) [noise=true , address=64.49.13.140:4516 , piece_id=VQRJQW4MMFG3JVOKIA6IWBGTRJYW7WLTKPISOPZTYX5BUOCJMWLA, seg_pos={0 0}, upload=502959]: Done uploading piece. err: <nil>
Dec 29 13:10:57.020149136 (573h41m56.891232181s) [noise=true , address=145.79.227.3:4184 , piece_id=5CSG4AMBAEH6K6NAERPU7FYYIGPTAD5BX6LAEH6Z35JODTUNXGUQ, seg_pos={0 0}, upload=502959]: Done uploading piece. err: <nil>
Dec 29 13:10:57.296405678 (573h41m57.167488723s) [noise=true , address=77.90.31.18:4821 , piece_id=RWDYWW3YSVKD3TARAI5BOBXC6RM7PBQNNR44JZSJKOHBTNTGAU4Q, seg_pos={0 0}, upload=502959]: Done uploading piece. err: <nil>
Dec 29 13:10:57.301628784 (573h41m57.172711819s) [noise=true , address=77.90.31.14:4691 , piece_id=GTR3MK7AZIX3VFMZMJ5YERMNR5KTHI2LFE42MINU3HOU3RHMWK5A, seg_pos={0 0}, upload=502959]: Done uploading piece. err: <nil>
Dec 29 13:10:57.378139311 (573h41m57.249222356s) [noise=true , address=45.91.116.5:4327 , piece_id=VHAVSNFNQCBZAL77IF4Y665J5LT2PUIEGW7NZI26FGF6ZJCW24IA, seg_pos={0 0}, upload=502959]: Done uploading piece. err: <nil>
Dec 29 13:10:57.436830097 (573h41m57.307913142s) [noise=true , address=45.91.116.13:4175 , piece_id=BHAA4Y5PONUKIYP7ZCUQX4ALUNOIB57GTEVSI5DXKJWAH3YRZAVA, seg_pos={0 0}, upload=502959]: Done uploading piece. err: <nil>
Dec 29 13:10:57.850451423 (573h41m57.721534488s) [noise=true , address=193.149.164.143:4699 , piece_id=DOBA6ISR7ICDWOERBGVKIKPW4F7ZEYWZKJPVD6QAEFVWIGBZK6ZQ, seg_pos={0 0}, upload=502959]: Done uploading piece. err: <nil>
Dec 29 13:10:57.984926172 (573h41m57.856009207s) [noise=true , address=145.79.227.40:4053 , piece_id=2KH5524YHDP6XVLSF7MQHJV5XLB2SXJDMUQALREV7XDNGP7T3A3Q, seg_pos={0 0}, upload=502959]: Done uploading piece. err: <nil>
Dec 29 13:10:57.987747283 (573h41m57.858830318s) [noise=true , address=145.79.227.30:4778 , piece_id=4RVR62UZNMHZX7TTUER5L5E6R2NED47CDKLWUUUROPTM7KTLEOFA, seg_pos={0 0}, upload=502959]: Done uploading piece. err: <nil>
Dec 29 13:10:57.990802792 (573h41m57.861885827s) [noise=true , address=145.79.227.44:4197 , piece_id=SXSYUVWVWAH76UB23LW3IL2XTMRJEZAVT2BISLGMYRL6S3ZNIHAQ, seg_pos={0 0}, upload=502959]: Done uploading piece. err: <nil>
Dec 29 13:10:57.998223692 (573h41m57.869306727s) [noise=true , address=193.149.164.143:4694 , piece_id=OIW4FHJF7BESOLVLTUK5VOHSREK35T5M3SUKZHICW7XVN6CDW44Q, seg_pos={0 0}, upload=502959]: Done uploading piece. err: <nil>
Dec 29 13:10:58.104926870 (573h41m57.976009905s) [noise=true , address=145.79.227.49:4365 , piece_id=IBRPNRC5EUT5FPD55YWZ7E3VQWQZW3FIWQWYFW4WTEHKMAQKCLCQ, seg_pos={0 0}, upload=502959]: Done uploading piece. err: <nil>
Dec 29 13:10:58.149661410 (573h41m58.020744445s) [noise=true , address=77.90.31.21:4887 , piece_id=JCF2JXHL3IYZEGYJ5YR37ZEKT27S4H7PDYNFP2PUKML7UVGTWZIQ, seg_pos={0 0}, upload=502959]: Done uploading piece. err: <nil>
Dec 29 13:10:58.361082928 (573h41m58.232165963s) [noise=true , address=45.91.116.10:4112 , piece_id=2E6GESG6RHHBKPQNB2LBCMXQK2CR7N4DHGV4AQQUUMMCBBQVUGKQ, seg_pos={0 0}, upload=502959]: Done uploading piece. err: <nil>
Dec 29 13:10:58.361106463 (573h41m58.232189498s) [seg_pos={0 0}, upload=502959]: Segment reached optimal threshold of 14 pieces.
Dec 29 13:10:58.361319789 (573h41m58.232402824s) [noise=true , address=64.49.13.169:4711 , piece_id=CTQNJBLYSYQU6SC4QOCF4EQGSAPRC7OHU2ON63V6A7TNPIT3PZZA, seg_pos={0 0}, upload=502959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:54358->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:54358->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54358->64.49.13.169:4711: use of closed network connection
Dec 29 13:10:58.361328425 (573h41m58.23241146s) [noise=true , address=64.49.13.169:4711 , piece_id=CTQNJBLYSYQU6SC4QOCF4EQGSAPRC7OHU2ON63V6A7TNPIT3PZZA, seg_pos={0 0}, upload=502959]: Long tail cancellation due to optimal threshold
Dec 29 13:10:58.361507426 (573h41m58.232590461s) [noise=true , address=145.79.227.33:4872 , piece_id=375VDGMK3XNUHA5I6SXX66RSHWSBPRA2JLS3WGQNFGEWOHECLH3Q, seg_pos={0 0}, upload=502959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12dNdm2kAok8jFgbYE24oXUL9KCJVDDiinLUJgDSYEY3fTEBJec): protocol: expected piece hash; context canceled; EOF
Dec 29 13:10:58.361517556 (573h41m58.232600591s) [noise=true , address=145.79.227.33:4872 , piece_id=375VDGMK3XNUHA5I6SXX66RSHWSBPRA2JLS3WGQNFGEWOHECLH3Q, seg_pos={0 0}, upload=502959]: Long tail cancellation due to optimal threshold
Dec 29 13:10:58.361527415 (573h41m58.23261045s) [noise=true , address=77.90.31.13:4637 , piece_id=ZT7OZ4HBHYFQ6RNS3IJOVFPJAM7SXNF4CJYDETCMUQET2OQ7ISAA, seg_pos={0 0}, upload=502959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17): protocol: read tcp 188.166.225.170:48470->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 29 13:10:58.361537994 (573h41m58.232621029s) [noise=true , address=77.90.31.13:4637 , piece_id=ZT7OZ4HBHYFQ6RNS3IJOVFPJAM7SXNF4CJYDETCMUQET2OQ7ISAA, seg_pos={0 0}, upload=502959]: Long tail cancellation due to optimal threshold
Dec 29 13:10:58.361669184 (573h41m58.232752219s) [noise=true , address=145.79.227.43:4179 , piece_id=OCQGPIZZSXSRPIL7OVCY6BCAQRWJLFVYY3KNYQYUEMDEOCUXJJGA, seg_pos={0 0}, upload=502959]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gh7eXJMeRFyjBP2e9ZyTQVVE1mMoMFfSkQ9ssfxJyqG63C4gP): protocol: expected piece hash; context canceled; EOF
Dec 29 13:10:58.361675416 (573h41m58.232758451s) [noise=true , address=145.79.227.43:4179 , piece_id=OCQGPIZZSXSRPIL7OVCY6BCAQRWJLFVYY3KNYQYUEMDEOCUXJJGA, seg_pos={0 0}, upload=502959]: Long tail cancellation due to optimal threshold
Dec 29 13:10:58.361780636 (573h41m58.232863671s) [seg_pos={0 0}, upload=502959]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 13:10:58.361903511 (573h41m58.232986556s) [upload=502959]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:10:58.361909492 (573h41m58.232992527s) [upload=502959]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:10:58.662421039 (573h41m58.533504084s) [upload=502959]: Done uploading segments...
Dec 29 13:10:58.662439815 (573h41m58.533522851s) [upload=502959]: Upload finished. err: <nil>
Duration: 1m0.002190472s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
51 out of 60 is failed (threshold=6) [XXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXX.X.XX.XXX.XX.XXXXXXX.XX.XXXX] duration:
5.000606263s trace:
https://jaeger.dev.storj.io/trace/19693652ae3afb57
FAIL (2025-12-29 13:11:45 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000629477s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
51 out of 60 is failed (threshold=6) [XXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXX.X.XX.XXX.XX.XXXXXXX.XX.XXX] duration:
5.001067733s trace:
https://jaeger.dev.storj.io/trace/590e54ce24398dc6
FAIL (2025-12-29 13:11:24 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001085246s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
51 out of 60 is failed (threshold=6) [XXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXX.X.XX.XXX.XX.XXXXXXX.XX.XX] duration:
5.001229999s trace:
https://jaeger.dev.storj.io/trace/61335e3568f2bdb6
FAIL (2025-12-29 13:11:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001254025s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.........X......................X.....X.....................] duration:
36.081601706s trace:
https://jaeger.dev.storj.io/trace/213826fef0d4f1dd
FAIL (2025-12-29 13:10:53 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
c 29 12:52:02.421331209 (573h23m2.292414254s) [noise=true , address=145.79.227.48:4334 , piece_id=MFEYVH6L2DBOS7Z444QFOQDVQCQS5MG3M32335L7ZCN4MXVH5L2A, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.421480444 (573h23m2.292563479s) [noise=true , address=77.90.31.17:4799 , piece_id=DFRKPKCVCIPZZJB6QDU64E5JGJIIKXDEO5MJAA4X5V6N3ZTEGTFA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.422126223 (573h23m2.293209268s) [noise=true , address=77.90.31.14:4676 , piece_id=RGKFUGBFBMHXICXNPPCFXNFHOZ5EXIFPT54PBT2EA5AS2ZC5OTGA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.422527958 (573h23m2.293611013s) [noise=true , address=145.79.227.17:4980 , piece_id=D3CAP7TOKGQ2NVULNTN3WSB2OQKOLVWN7TFGLPOLLJPPMHHZR7NQ, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.422668796 (573h23m2.293751841s) [noise=true , address=193.149.164.143:4697 , piece_id=C2JF73NKW64HNXGS5BJTVZRCAIVWKZBGDGUZFJMDUWW3RBBXJDSA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.422966322 (573h23m2.294049357s) [noise=true , address=77.90.31.19:4863 , piece_id=XLK3SHF2GNV625H7RC5SGL67TKKBAJREGPTBCEJDX3QGD6H63GMA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.423193946 (573h23m2.294276981s) [noise=true , address=77.90.31.18:4829 , piece_id=RG37QNRSJFFED3XOPAEOHGJNJ45MEMUON3R3MB5ASEXS4A4EQ42Q, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.423513825 (573h23m2.29459687s) [noise=true , address=145.79.227.43:4166 , piece_id=QPEP6OT7WAFMRNNI6XVL25H3FAJ76FCQXVSEACOTV6XCYIZZSVPA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.423810040 (573h23m2.294893085s) [noise=true , address=77.90.31.10:4545 , piece_id=NE3YEJJJKKOQSM2JKO5PFTPJH7SG65UZGR3QJP5JZXOUF3FN32YQ, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.424441803 (573h23m2.295524868s) [noise=true , address=64.49.13.169:4708 , piece_id=5ORIIBIW2GPJRDD5C7IBQDPKPS5SJBTSVR6SGC6NKX36DTTGAYGA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.425051363 (573h23m2.296134408s) [noise=true , address=45.91.116.5:4323 , piece_id=7FAJKQP23A6U6SXPWVVAFVIHQA54JSPERWMCSONK5NYNHXRDLF3A, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.425681483 (573h23m2.296764528s) [noise=true , address=145.79.227.7:4303 , piece_id=MWIQ7KUZ534TFKQFMQADMBPKUFDGNFFK3JJEH3KF2CT6IQGHQRKQ, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.426480093 (573h23m2.297563138s) [noise=true , address=145.79.227.33:4877 , piece_id=IFC4ANBQU7ORCMSTYAAG3QKE4S2JPCWKDD47P5Z57V6TWAR6GFTA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.427077340 (573h23m2.298160385s) [noise=true , address=145.79.227.34:4937 , piece_id=2EYFRE3YDBIZK7YORZMWRETOEHJGDDKJVZBFW36FU6LX6PHRTDIA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.427529692 (573h23m2.298612737s) [noise=true , address=145.79.227.30:4793 , piece_id=AGUHSTJNNNRAAP5TKWIONYBZDKPJDJEQX7RX5GEGWN6TG4RELAMQ, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.428130125 (573h23m2.29921317s) [noise=true , address=77.90.31.16:4747 , piece_id=5TQY6C4SF6JQT6KTSEMOSY65TLPLUVCDTNVJ6U2QYFMTYP2G73VA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:03.783583605 (573h23m3.65466665s) [upload=502680]: Next returned nil segment
Dec 29 12:52:03.783597080 (573h23m3.654680125s) [upload=502680]: Waiting for error group managing segments...
Dec 29 12:52:04.132513046 (573h23m4.003596091s) [noise=true , address=64.49.13.137:4494 , piece_id=K4TIPJCGNFYIMSQKXQAWABJLIHHTAS3F36I723BHTJ7YG4HUKIIQ, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.271420967 (573h23m4.142504012s) [noise=true , address=145.79.227.7:4303 , piece_id=MWIQ7KUZ534TFKQFMQADMBPKUFDGNFFK3JJEH3KF2CT6IQGHQRKQ, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.275668766 (573h23m4.146751811s) [noise=true , address=145.79.227.48:4334 , piece_id=MFEYVH6L2DBOS7Z444QFOQDVQCQS5MG3M32335L7ZCN4MXVH5L2A, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.298256927 (573h23m4.169339972s) [noise=true , address=145.79.227.17:4980 , piece_id=D3CAP7TOKGQ2NVULNTN3WSB2OQKOLVWN7TFGLPOLLJPPMHHZR7NQ, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.487163872 (573h23m4.358246917s) [noise=true , address=77.90.31.19:4863 , piece_id=XLK3SHF2GNV625H7RC5SGL67TKKBAJREGPTBCEJDX3QGD6H63GMA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.563458298 (573h23m4.434541343s) [noise=true , address=45.91.116.5:4323 , piece_id=7FAJKQP23A6U6SXPWVVAFVIHQA54JSPERWMCSONK5NYNHXRDLF3A, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.572838760 (573h23m4.443921795s) [noise=true , address=77.90.31.10:4545 , piece_id=NE3YEJJJKKOQSM2JKO5PFTPJH7SG65UZGR3QJP5JZXOUF3FN32YQ, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.580866937 (573h23m4.451949972s) [noise=true , address=77.90.31.14:4676 , piece_id=RGKFUGBFBMHXICXNPPCFXNFHOZ5EXIFPT54PBT2EA5AS2ZC5OTGA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.595018524 (573h23m4.466101569s) [noise=true , address=77.90.31.17:4799 , piece_id=DFRKPKCVCIPZZJB6QDU64E5JGJIIKXDEO5MJAA4X5V6N3ZTEGTFA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:05.183831178 (573h23m5.054914213s) [noise=true , address=145.79.227.30:4793 , piece_id=AGUHSTJNNNRAAP5TKWIONYBZDKPJDJEQX7RX5GEGWN6TG4RELAMQ, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:05.242293548 (573h23m5.113376593s) [noise=true , address=145.79.227.33:4877 , piece_id=IFC4ANBQU7ORCMSTYAAG3QKE4S2JPCWKDD47P5Z57V6TWAR6GFTA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:05.255836156 (573h23m5.126919201s) [noise=true , address=145.79.227.34:4937 , piece_id=2EYFRE3YDBIZK7YORZMWRETOEHJGDDKJVZBFW36FU6LX6PHRTDIA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:05.305203848 (573h23m5.176286883s) [noise=true , address=145.79.227.40:4058 , piece_id=EN2T5WMAEV3PVNAOWLMN4BUIJSAXRMAIQBQ6PKQ7PWPATZLYYBIQ, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:05.326720429 (573h23m5.197803474s) [noise=true , address=145.79.227.43:4166 , piece_id=QPEP6OT7WAFMRNNI6XVL25H3FAJ76FCQXVSEACOTV6XCYIZZSVPA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:05.326747741 (573h23m5.197830786s) [seg_pos={0 0}, upload=502680]: Segment reached optimal threshold of 14 pieces.
Dec 29 12:52:05.327021833 (573h23m5.198104889s) [noise=true , address=77.90.31.18:4829 , piece_id=RG37QNRSJFFED3XOPAEOHGJNJ45MEMUON3R3MB5ASEXS4A4EQ42Q, seg_pos={0 0}, upload=502680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo): protocol: expected piece hash; context canceled; EOF
Dec 29 12:52:05.327031712 (573h23m5.198114757s) [noise=true , address=77.90.31.18:4829 , piece_id=RG37QNRSJFFED3XOPAEOHGJNJ45MEMUON3R3MB5ASEXS4A4EQ42Q, seg_pos={0 0}, upload=502680]: Long tail cancellation due to optimal threshold
Dec 29 12:52:05.327058794 (573h23m5.198141849s) [noise=true , address=64.49.13.169:4708 , piece_id=5ORIIBIW2GPJRDD5C7IBQDPKPS5SJBTSVR6SGC6NKX36DTTGAYGA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:40464->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:40464->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40464->64.49.13.169:4708: use of closed network connection
Dec 29 12:52:05.327066769 (573h23m5.198149814s) [noise=true , address=64.49.13.169:4708 , piece_id=5ORIIBIW2GPJRDD5C7IBQDPKPS5SJBTSVR6SGC6NKX36DTTGAYGA, seg_pos={0 0}, upload=502680]: Long tail cancellation due to optimal threshold
Dec 29 12:52:05.327203149 (573h23m5.198286204s) [noise=true , address=77.90.31.16:4747 , piece_id=5TQY6C4SF6JQT6KTSEMOSY65TLPLUVCDTNVJ6U2QYFMTYP2G73VA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2): protocol: expected piece hash; context canceled; EOF
Dec 29 12:52:05.327213137 (573h23m5.198296182s) [noise=true , address=77.90.31.16:4747 , piece_id=5TQY6C4SF6JQT6KTSEMOSY65TLPLUVCDTNVJ6U2QYFMTYP2G73VA, seg_pos={0 0}, upload=502680]: Long tail cancellation due to optimal threshold
Dec 29 12:52:05.327491227 (573h23m5.198574282s) [noise=true , address=193.149.164.143:4697 , piece_id=C2JF73NKW64HNXGS5BJTVZRCAIVWKZBGDGUZFJMDUWW3RBBXJDSA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF
Dec 29 12:52:05.327500265 (573h23m5.1985833s) [noise=true , address=193.149.164.143:4697 , piece_id=C2JF73NKW64HNXGS5BJTVZRCAIVWKZBGDGUZFJMDUWW3RBBXJDSA, seg_pos={0 0}, upload=502680]: Long tail cancellation due to optimal threshold
Dec 29 12:52:05.327548276 (573h23m5.198631321s) [seg_pos={0 0}, upload=502680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 12:52:05.327736143 (573h23m5.198819188s) [upload=502680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 12:52:05.327744139 (573h23m5.198827174s) [upload=502680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 12:52:05.624748327 (573h23m5.495831362s) [upload=502680]: Done uploading segments...
Dec 29 12:52:05.624772935 (573h23m5.49585597s) [upload=502680]: Upload finished. err: <nil>
Duration: 36.081690845s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.........................X......................X....X.X....] duration:
13.784069797s trace:
https://jaeger.dev.storj.io/trace/141b6edfc3103b12
FAIL (2025-12-29 13:10:53 UTC)
orbiter warning: context deadline exceeded
Duration: 13.784095687s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
29 out of 60 is failed (threshold=3) [..X..X....XXXX..XX..X...XX.X.XX...X.X.X.XXX.X.X.X.XX..X.XXX.] duration:
6.030443477s trace:
https://jaeger.dev.storj.io/trace/43ee0528f7a7db89
FAIL (2025-12-29 13:10:52 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
29 13:10:16.525384466 (573h41m16.396467521s) [upload=502950]: Starting upload
Dec 29 13:10:16.525470861 (573h41m16.396553896s) [upload=502950]: Done starting upload
Dec 29 13:10:16.525691761 (573h41m16.396774806s) [upload=502950]: Uploading segments...
Dec 29 13:10:16.525834794 (573h41m16.396917839s) [upload=502950]: Got next segment. Inline: false
Dec 29 13:10:16.525875110 (573h41m16.396958145s) [upload=502950]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 13:10:16.525882896 (573h41m16.396965941s) [upload=502950]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 13:10:16.771726077 (573h41m16.642809122s) [seg_pos={0 0}, upload=502950]: Begin upload segment...
Dec 29 13:10:16.771781223 (573h41m16.642864268s) [seg_pos={0 0}, upload=502950]: Done begin upload segment.
Dec 29 13:10:16.771813534 (573h41m16.642896579s) [noise=true , address=145.79.227.40:4050 , piece_id=62XCMIIDXGVHPKQQZ6YGADIBS5DN3F2F3O3OF22Z5EYJD5QCWZJA, seg_pos={0 0}, upload=502950]: Uploading piece...
Dec 29 13:10:16.771991193 (573h41m16.643074238s) [noise=true , address=145.79.227.4:4187 , piece_id=NJHSK6QJTREZTBTXNYD4IBV2QX3MZVI7TIEUTV2QL4TE74GKYZVQ, seg_pos={0 0}, upload=502950]: Uploading piece...
Dec 29 13:10:16.772617595 (573h41m16.64370065s) [noise=true , address=145.79.227.15:4875 , piece_id=7ZWQAAAVAYSCC4ZT5HXBDG43GYRIYOJWV5BM5AXEGGRPBDX6HLJA, seg_pos={0 0}, upload=502950]: Uploading piece...
Dec 29 13:10:16.773080587 (573h41m16.644163632s) [noise=true , address=77.90.31.11:4555 , piece_id=H36QZB5JLJ7VRV3WJB23A23RPC5YKUIDKYESIEGS6GN7UZZYLKZQ, seg_pos={0 0}, upload=502950]: Uploading piece...
Dec 29 13:10:16.773528430 (573h41m16.644611475s) [noise=true , address=145.79.227.3:4151 , piece_id=H4KFFVVKKJA7JLVAFQMW5A4TONUXWPYUVGJODOQHEZHYRMMHSWYQ, seg_pos={0 0}, upload=502950]: Uploading piece...
Dec 29 13:10:16.774033290 (573h41m16.645116336s) [noise=true , address=145.79.227.10:4401 , piece_id=EWTCSLUZGRNQI7IVTN6VPSAZKFRE6OERIWYSTZHGBF3Q7VHFQRNQ, seg_pos={0 0}, upload=502950]: Uploading piece...
Dec 29 13:10:16.774635989 (573h41m16.645719034s) [noise=true , address=145.79.227.37:4567 , piece_id=6O54TQ5VJ4XDXWB3NLRO4DNVYCUSTDQOP4YN5E3SBX2W2SJ2VYWQ, seg_pos={0 0}, upload=502950]: Uploading piece...
Dec 29 13:10:16.775114469 (573h41m16.646197514s) [noise=true , address=145.79.227.19:4077 , piece_id=WEOZ4ITE3ZDJSPG2QMJ6MWU2B6U47SFOXHLXIFLT6HAGOWZAFSLA, seg_pos={0 0}, upload=502950]: Uploading piece...
Dec 29 13:10:16.776789348 (573h41m16.647872393s) [noise=true , address=145.79.227.45:4231 , piece_id=TVLWB4ZDELBLYNFL5WMVQZ5KWHQXT3WL5CIPPEGPGJRLZXVAZSDQ, seg_pos={0 0}, upload=502950]: Uploading piece...
Dec 29 13:10:16.776814046 (573h41m16.647897091s) [noise=true , address=145.79.227.42:4121 , piece_id=HIVAZJQK36HXOLTLE3ERLOZB3EBWAYOVZMLF6EIWGQFZ5VRC5JSQ, seg_pos={0 0}, upload=502950]: Uploading piece...
Dec 29 13:10:16.778580550 (573h41m16.649663585s) [noise=true , address=145.79.227.5:4222 , piece_id=BSMZITIJPCCGAZNWRYXZY4M2WECJQW4NIB6YLQHAFWKIW5ZWVO4A, seg_pos={0 0}, upload=502950]: Uploading piece...
Dec 29 13:10:16.781783197 (573h41m16.652866242s) [noise=true , address=45.91.116.5:4298 , piece_id=WTI3ZUYE4MSPWUSAQ4NNLX6FX56L2UYOVI4R7TAICUCP2WZPAMUA, seg_pos={0 0}, upload=502950]: Uploading piece...
Dec 29 13:10:16.783583846 (573h41m16.654666881s) [noise=true , address=45.91.116.13:4155 , piece_id=UQBEV3RPRIJCTEKMQH7MJZMJ6AS4GZRGDEOAFG22N2TDU7IU35EQ, seg_pos={0 0}, upload=502950]: Uploading piece...
Dec 29 13:10:16.785133738 (573h41m16.656216783s) [noise=true , address=145.79.227.11:4731 , piece_id=ZTISKKPLAZS4P43G7QO24OHAKXJVMERGFKICDLPH734HUGZPY3TQ, seg_pos={0 0}, upload=502950]: Uploading piece...
Dec 29 13:10:16.786744705 (573h41m16.65782774s) [noise=true , address=45.91.116.14:4192 , piece_id=WALXEJIUR2XJ4INKVENUJXA4WOVJUKXIB466KBSI2O3J7OBXP56A, seg_pos={0 0}, upload=502950]: Uploading piece...
Dec 29 13:10:16.788263487 (573h41m16.659346532s) [noise=true , address=145.79.227.47:4266 , piece_id=J6DIEO4TI4OX7VBABG6YP3SUHCKHX7HZ4UU4VKV6QZDVP7VCDSEA, seg_pos={0 0}, upload=502950]: Uploading piece...
Dec 29 13:10:18.391241591 (573h41m18.262324636s) [upload=502950]: Next returned nil segment
Dec 29 13:10:18.391258172 (573h41m18.262341217s) [upload=502950]: Waiting for error group managing segments...
Dec 29 13:10:18.772110892 (573h41m18.643193937s) [noise=true , address=145.79.227.11:4731 , piece_id=ZTISKKPLAZS4P43G7QO24OHAKXJVMERGFKICDLPH734HUGZPY3TQ, seg_pos={0 0}, upload=502950]: Done uploading piece. err: <nil>
Dec 29 13:10:18.772522857 (573h41m18.643605892s) [noise=true , address=145.79.227.4:4187 , piece_id=NJHSK6QJTREZTBTXNYD4IBV2QX3MZVI7TIEUTV2QL4TE74GKYZVQ, seg_pos={0 0}, upload=502950]: Done uploading piece. err: <nil>
Dec 29 13:10:18.773631066 (573h41m18.644714111s) [noise=true , address=145.79.227.3:4151 , piece_id=H4KFFVVKKJA7JLVAFQMW5A4TONUXWPYUVGJODOQHEZHYRMMHSWYQ, seg_pos={0 0}, upload=502950]: Done uploading piece. err: <nil>
Dec 29 13:10:18.773899317 (573h41m18.644982362s) [noise=true , address=145.79.227.19:4077 , piece_id=WEOZ4ITE3ZDJSPG2QMJ6MWU2B6U47SFOXHLXIFLT6HAGOWZAFSLA, seg_pos={0 0}, upload=502950]: Done uploading piece. err: <nil>
Dec 29 13:10:18.779429688 (573h41m18.650512743s) [noise=true , address=145.79.227.15:4875 , piece_id=7ZWQAAAVAYSCC4ZT5HXBDG43GYRIYOJWV5BM5AXEGGRPBDX6HLJA, seg_pos={0 0}, upload=502950]: Done uploading piece. err: <nil>
Dec 29 13:10:18.788714718 (573h41m18.659797763s) [noise=true , address=145.79.227.47:4266 , piece_id=J6DIEO4TI4OX7VBABG6YP3SUHCKHX7HZ4UU4VKV6QZDVP7VCDSEA, seg_pos={0 0}, upload=502950]: Done uploading piece. err: <nil>
Dec 29 13:10:18.788998688 (573h41m18.660081733s) [noise=true , address=145.79.227.37:4567 , piece_id=6O54TQ5VJ4XDXWB3NLRO4DNVYCUSTDQOP4YN5E3SBX2W2SJ2VYWQ, seg_pos={0 0}, upload=502950]: Done uploading piece. err: <nil>
Dec 29 13:10:18.789608009 (573h41m18.660691054s) [noise=true , address=145.79.227.10:4401 , piece_id=EWTCSLUZGRNQI7IVTN6VPSAZKFRE6OERIWYSTZHGBF3Q7VHFQRNQ, seg_pos={0 0}, upload=502950]: Done uploading piece. err: <nil>
Dec 29 13:10:18.789863445 (573h41m18.66094648s) [noise=true , address=145.79.227.42:4121 , piece_id=HIVAZJQK36HXOLTLE3ERLOZB3EBWAYOVZMLF6EIWGQFZ5VRC5JSQ, seg_pos={0 0}, upload=502950]: Done uploading piece. err: <nil>
Dec 29 13:10:18.795057986 (573h41m18.666141031s) [noise=true , address=145.79.227.5:4222 , piece_id=BSMZITIJPCCGAZNWRYXZY4M2WECJQW4NIB6YLQHAFWKIW5ZWVO4A, seg_pos={0 0}, upload=502950]: Done uploading piece. err: <nil>
Dec 29 13:10:18.795365011 (573h41m18.666448056s) [noise=true , address=145.79.227.45:4231 , piece_id=TVLWB4ZDELBLYNFL5WMVQZ5KWHQXT3WL5CIPPEGPGJRLZXVAZSDQ, seg_pos={0 0}, upload=502950]: Done uploading piece. err: <nil>
Dec 29 13:10:19.099709147 (573h41m18.970792192s) [noise=true , address=45.91.116.5:4298 , piece_id=WTI3ZUYE4MSPWUSAQ4NNLX6FX56L2UYOVI4R7TAICUCP2WZPAMUA, seg_pos={0 0}, upload=502950]: Done uploading piece. err: <nil>
Dec 29 13:10:19.129290354 (573h41m19.000373389s) [noise=true , address=45.91.116.13:4155 , piece_id=UQBEV3RPRIJCTEKMQH7MJZMJ6AS4GZRGDEOAFG22N2TDU7IU35EQ, seg_pos={0 0}, upload=502950]: Done uploading piece. err: <nil>
Dec 29 13:10:19.129316303 (573h41m19.000399338s) [seg_pos={0 0}, upload=502950]: Segment reached optimal threshold of 13 pieces.
Dec 29 13:10:19.129521224 (573h41m19.000604269s) [noise=true , address=45.91.116.14:4192 , piece_id=WALXEJIUR2XJ4INKVENUJXA4WOVJUKXIB466KBSI2O3J7OBXP56A, seg_pos={0 0}, upload=502950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF
Dec 29 13:10:19.129533938 (573h41m19.000616983s) [noise=true , address=45.91.116.14:4192 , piece_id=WALXEJIUR2XJ4INKVENUJXA4WOVJUKXIB466KBSI2O3J7OBXP56A, seg_pos={0 0}, upload=502950]: Long tail cancellation due to optimal threshold
Dec 29 13:10:19.129544698 (573h41m19.000627743s) [noise=true , address=145.79.227.40:4050 , piece_id=62XCMIIDXGVHPKQQZ6YGADIBS5DN3F2F3O3OF22Z5EYJD5QCWZJA, seg_pos={0 0}, upload=502950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF
Dec 29 13:10:19.129549508 (573h41m19.000632543s) [noise=true , address=145.79.227.40:4050 , piece_id=62XCMIIDXGVHPKQQZ6YGADIBS5DN3F2F3O3OF22Z5EYJD5QCWZJA, seg_pos={0 0}, upload=502950]: Long tail cancellation due to optimal threshold
Dec 29 13:10:19.129721995 (573h41m19.00080504s) [noise=true , address=77.90.31.11:4555 , piece_id=H36QZB5JLJ7VRV3WJB23A23RPC5YKUIDKYESIEGS6GN7UZZYLKZQ, seg_pos={0 0}, upload=502950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: write tcp 188.166.225.170:33118->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:33118->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33118->77.90.31.11:4555: use of closed network connection
Dec 29 13:10:19.129732576 (573h41m19.000815631s) [noise=true , address=77.90.31.11:4555 , piece_id=H36QZB5JLJ7VRV3WJB23A23RPC5YKUIDKYESIEGS6GN7UZZYLKZQ, seg_pos={0 0}, upload=502950]: Long tail cancellation due to optimal threshold
Dec 29 13:10:19.129753475 (573h41m19.00083652s) [seg_pos={0 0}, upload=502950]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 13:10:19.129876710 (573h41m19.000959755s) [upload=502950]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:10:19.129883924 (573h41m19.000966969s) [upload=502950]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:10:19.419583054 (573h41m19.290666119s) [upload=502950]: Done uploading segments...
Dec 29 13:10:19.419614354 (573h41m19.290697399s) [upload=502950]: Upload finished. err: <nil>
Duration: 6.030457694s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
51 out of 60 is failed (threshold=6) [XXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXX.X.XX.XXX.XX.XXXXXXX.XX.X] duration:
5.001305607s trace:
https://jaeger.dev.storj.io/trace/561d1353aee5c4e4
FAIL (2025-12-29 13:10:49 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001327579s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
29 out of 60 is failed (threshold=3) [...X..X....XXXX..XX..X...XX.X.XX...X.X.X.XXX.X.X.X.XX..X.XXX] duration:
30.00239756s trace:
https://jaeger.dev.storj.io/trace/43ee0528f7a7db89
FAIL (2025-12-29 13:10:45 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
29 13:10:16.525384466 (573h41m16.396467521s) [upload=502950]: Starting upload
Dec 29 13:10:16.525470861 (573h41m16.396553896s) [upload=502950]: Done starting upload
Dec 29 13:10:16.525691761 (573h41m16.396774806s) [upload=502950]: Uploading segments...
Dec 29 13:10:16.525834794 (573h41m16.396917839s) [upload=502950]: Got next segment. Inline: false
Dec 29 13:10:16.525875110 (573h41m16.396958145s) [upload=502950]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 13:10:16.525882896 (573h41m16.396965941s) [upload=502950]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 13:10:16.771726077 (573h41m16.642809122s) [seg_pos={0 0}, upload=502950]: Begin upload segment...
Dec 29 13:10:16.771781223 (573h41m16.642864268s) [seg_pos={0 0}, upload=502950]: Done begin upload segment.
Dec 29 13:10:16.771813534 (573h41m16.642896579s) [noise=true , address=145.79.227.40:4050 , piece_id=62XCMIIDXGVHPKQQZ6YGADIBS5DN3F2F3O3OF22Z5EYJD5QCWZJA, seg_pos={0 0}, upload=502950]: Uploading piece...
Dec 29 13:10:16.771991193 (573h41m16.643074238s) [noise=true , address=145.79.227.4:4187 , piece_id=NJHSK6QJTREZTBTXNYD4IBV2QX3MZVI7TIEUTV2QL4TE74GKYZVQ, seg_pos={0 0}, upload=502950]: Uploading piece...
Dec 29 13:10:16.772617595 (573h41m16.64370065s) [noise=true , address=145.79.227.15:4875 , piece_id=7ZWQAAAVAYSCC4ZT5HXBDG43GYRIYOJWV5BM5AXEGGRPBDX6HLJA, seg_pos={0 0}, upload=502950]: Uploading piece...
Dec 29 13:10:16.773080587 (573h41m16.644163632s) [noise=true , address=77.90.31.11:4555 , piece_id=H36QZB5JLJ7VRV3WJB23A23RPC5YKUIDKYESIEGS6GN7UZZYLKZQ, seg_pos={0 0}, upload=502950]: Uploading piece...
Dec 29 13:10:16.773528430 (573h41m16.644611475s) [noise=true , address=145.79.227.3:4151 , piece_id=H4KFFVVKKJA7JLVAFQMW5A4TONUXWPYUVGJODOQHEZHYRMMHSWYQ, seg_pos={0 0}, upload=502950]: Uploading piece...
Dec 29 13:10:16.774033290 (573h41m16.645116336s) [noise=true , address=145.79.227.10:4401 , piece_id=EWTCSLUZGRNQI7IVTN6VPSAZKFRE6OERIWYSTZHGBF3Q7VHFQRNQ, seg_pos={0 0}, upload=502950]: Uploading piece...
Dec 29 13:10:16.774635989 (573h41m16.645719034s) [noise=true , address=145.79.227.37:4567 , piece_id=6O54TQ5VJ4XDXWB3NLRO4DNVYCUSTDQOP4YN5E3SBX2W2SJ2VYWQ, seg_pos={0 0}, upload=502950]: Uploading piece...
Dec 29 13:10:16.775114469 (573h41m16.646197514s) [noise=true , address=145.79.227.19:4077 , piece_id=WEOZ4ITE3ZDJSPG2QMJ6MWU2B6U47SFOXHLXIFLT6HAGOWZAFSLA, seg_pos={0 0}, upload=502950]: Uploading piece...
Dec 29 13:10:16.776789348 (573h41m16.647872393s) [noise=true , address=145.79.227.45:4231 , piece_id=TVLWB4ZDELBLYNFL5WMVQZ5KWHQXT3WL5CIPPEGPGJRLZXVAZSDQ, seg_pos={0 0}, upload=502950]: Uploading piece...
Dec 29 13:10:16.776814046 (573h41m16.647897091s) [noise=true , address=145.79.227.42:4121 , piece_id=HIVAZJQK36HXOLTLE3ERLOZB3EBWAYOVZMLF6EIWGQFZ5VRC5JSQ, seg_pos={0 0}, upload=502950]: Uploading piece...
Dec 29 13:10:16.778580550 (573h41m16.649663585s) [noise=true , address=145.79.227.5:4222 , piece_id=BSMZITIJPCCGAZNWRYXZY4M2WECJQW4NIB6YLQHAFWKIW5ZWVO4A, seg_pos={0 0}, upload=502950]: Uploading piece...
Dec 29 13:10:16.781783197 (573h41m16.652866242s) [noise=true , address=45.91.116.5:4298 , piece_id=WTI3ZUYE4MSPWUSAQ4NNLX6FX56L2UYOVI4R7TAICUCP2WZPAMUA, seg_pos={0 0}, upload=502950]: Uploading piece...
Dec 29 13:10:16.783583846 (573h41m16.654666881s) [noise=true , address=45.91.116.13:4155 , piece_id=UQBEV3RPRIJCTEKMQH7MJZMJ6AS4GZRGDEOAFG22N2TDU7IU35EQ, seg_pos={0 0}, upload=502950]: Uploading piece...
Dec 29 13:10:16.785133738 (573h41m16.656216783s) [noise=true , address=145.79.227.11:4731 , piece_id=ZTISKKPLAZS4P43G7QO24OHAKXJVMERGFKICDLPH734HUGZPY3TQ, seg_pos={0 0}, upload=502950]: Uploading piece...
Dec 29 13:10:16.786744705 (573h41m16.65782774s) [noise=true , address=45.91.116.14:4192 , piece_id=WALXEJIUR2XJ4INKVENUJXA4WOVJUKXIB466KBSI2O3J7OBXP56A, seg_pos={0 0}, upload=502950]: Uploading piece...
Dec 29 13:10:16.788263487 (573h41m16.659346532s) [noise=true , address=145.79.227.47:4266 , piece_id=J6DIEO4TI4OX7VBABG6YP3SUHCKHX7HZ4UU4VKV6QZDVP7VCDSEA, seg_pos={0 0}, upload=502950]: Uploading piece...
Dec 29 13:10:18.391241591 (573h41m18.262324636s) [upload=502950]: Next returned nil segment
Dec 29 13:10:18.391258172 (573h41m18.262341217s) [upload=502950]: Waiting for error group managing segments...
Dec 29 13:10:18.772110892 (573h41m18.643193937s) [noise=true , address=145.79.227.11:4731 , piece_id=ZTISKKPLAZS4P43G7QO24OHAKXJVMERGFKICDLPH734HUGZPY3TQ, seg_pos={0 0}, upload=502950]: Done uploading piece. err: <nil>
Dec 29 13:10:18.772522857 (573h41m18.643605892s) [noise=true , address=145.79.227.4:4187 , piece_id=NJHSK6QJTREZTBTXNYD4IBV2QX3MZVI7TIEUTV2QL4TE74GKYZVQ, seg_pos={0 0}, upload=502950]: Done uploading piece. err: <nil>
Dec 29 13:10:18.773631066 (573h41m18.644714111s) [noise=true , address=145.79.227.3:4151 , piece_id=H4KFFVVKKJA7JLVAFQMW5A4TONUXWPYUVGJODOQHEZHYRMMHSWYQ, seg_pos={0 0}, upload=502950]: Done uploading piece. err: <nil>
Dec 29 13:10:18.773899317 (573h41m18.644982362s) [noise=true , address=145.79.227.19:4077 , piece_id=WEOZ4ITE3ZDJSPG2QMJ6MWU2B6U47SFOXHLXIFLT6HAGOWZAFSLA, seg_pos={0 0}, upload=502950]: Done uploading piece. err: <nil>
Dec 29 13:10:18.779429688 (573h41m18.650512743s) [noise=true , address=145.79.227.15:4875 , piece_id=7ZWQAAAVAYSCC4ZT5HXBDG43GYRIYOJWV5BM5AXEGGRPBDX6HLJA, seg_pos={0 0}, upload=502950]: Done uploading piece. err: <nil>
Dec 29 13:10:18.788714718 (573h41m18.659797763s) [noise=true , address=145.79.227.47:4266 , piece_id=J6DIEO4TI4OX7VBABG6YP3SUHCKHX7HZ4UU4VKV6QZDVP7VCDSEA, seg_pos={0 0}, upload=502950]: Done uploading piece. err: <nil>
Dec 29 13:10:18.788998688 (573h41m18.660081733s) [noise=true , address=145.79.227.37:4567 , piece_id=6O54TQ5VJ4XDXWB3NLRO4DNVYCUSTDQOP4YN5E3SBX2W2SJ2VYWQ, seg_pos={0 0}, upload=502950]: Done uploading piece. err: <nil>
Dec 29 13:10:18.789608009 (573h41m18.660691054s) [noise=true , address=145.79.227.10:4401 , piece_id=EWTCSLUZGRNQI7IVTN6VPSAZKFRE6OERIWYSTZHGBF3Q7VHFQRNQ, seg_pos={0 0}, upload=502950]: Done uploading piece. err: <nil>
Dec 29 13:10:18.789863445 (573h41m18.66094648s) [noise=true , address=145.79.227.42:4121 , piece_id=HIVAZJQK36HXOLTLE3ERLOZB3EBWAYOVZMLF6EIWGQFZ5VRC5JSQ, seg_pos={0 0}, upload=502950]: Done uploading piece. err: <nil>
Dec 29 13:10:18.795057986 (573h41m18.666141031s) [noise=true , address=145.79.227.5:4222 , piece_id=BSMZITIJPCCGAZNWRYXZY4M2WECJQW4NIB6YLQHAFWKIW5ZWVO4A, seg_pos={0 0}, upload=502950]: Done uploading piece. err: <nil>
Dec 29 13:10:18.795365011 (573h41m18.666448056s) [noise=true , address=145.79.227.45:4231 , piece_id=TVLWB4ZDELBLYNFL5WMVQZ5KWHQXT3WL5CIPPEGPGJRLZXVAZSDQ, seg_pos={0 0}, upload=502950]: Done uploading piece. err: <nil>
Dec 29 13:10:19.099709147 (573h41m18.970792192s) [noise=true , address=45.91.116.5:4298 , piece_id=WTI3ZUYE4MSPWUSAQ4NNLX6FX56L2UYOVI4R7TAICUCP2WZPAMUA, seg_pos={0 0}, upload=502950]: Done uploading piece. err: <nil>
Dec 29 13:10:19.129290354 (573h41m19.000373389s) [noise=true , address=45.91.116.13:4155 , piece_id=UQBEV3RPRIJCTEKMQH7MJZMJ6AS4GZRGDEOAFG22N2TDU7IU35EQ, seg_pos={0 0}, upload=502950]: Done uploading piece. err: <nil>
Dec 29 13:10:19.129316303 (573h41m19.000399338s) [seg_pos={0 0}, upload=502950]: Segment reached optimal threshold of 13 pieces.
Dec 29 13:10:19.129521224 (573h41m19.000604269s) [noise=true , address=45.91.116.14:4192 , piece_id=WALXEJIUR2XJ4INKVENUJXA4WOVJUKXIB466KBSI2O3J7OBXP56A, seg_pos={0 0}, upload=502950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF
Dec 29 13:10:19.129533938 (573h41m19.000616983s) [noise=true , address=45.91.116.14:4192 , piece_id=WALXEJIUR2XJ4INKVENUJXA4WOVJUKXIB466KBSI2O3J7OBXP56A, seg_pos={0 0}, upload=502950]: Long tail cancellation due to optimal threshold
Dec 29 13:10:19.129544698 (573h41m19.000627743s) [noise=true , address=145.79.227.40:4050 , piece_id=62XCMIIDXGVHPKQQZ6YGADIBS5DN3F2F3O3OF22Z5EYJD5QCWZJA, seg_pos={0 0}, upload=502950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF
Dec 29 13:10:19.129549508 (573h41m19.000632543s) [noise=true , address=145.79.227.40:4050 , piece_id=62XCMIIDXGVHPKQQZ6YGADIBS5DN3F2F3O3OF22Z5EYJD5QCWZJA, seg_pos={0 0}, upload=502950]: Long tail cancellation due to optimal threshold
Dec 29 13:10:19.129721995 (573h41m19.00080504s) [noise=true , address=77.90.31.11:4555 , piece_id=H36QZB5JLJ7VRV3WJB23A23RPC5YKUIDKYESIEGS6GN7UZZYLKZQ, seg_pos={0 0}, upload=502950]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: write tcp 188.166.225.170:33118->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:33118->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33118->77.90.31.11:4555: use of closed network connection
Dec 29 13:10:19.129732576 (573h41m19.000815631s) [noise=true , address=77.90.31.11:4555 , piece_id=H36QZB5JLJ7VRV3WJB23A23RPC5YKUIDKYESIEGS6GN7UZZYLKZQ, seg_pos={0 0}, upload=502950]: Long tail cancellation due to optimal threshold
Dec 29 13:10:19.129753475 (573h41m19.00083652s) [seg_pos={0 0}, upload=502950]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 13:10:19.129876710 (573h41m19.000959755s) [upload=502950]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:10:19.129883924 (573h41m19.000966969s) [upload=502950]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:10:19.419583054 (573h41m19.290666119s) [upload=502950]: Done uploading segments...
Dec 29 13:10:19.419614354 (573h41m19.290697399s) [upload=502950]: Upload finished. err: <nil>
Duration: 30.002409011s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
51 out of 60 is failed (threshold=6) [XXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXX.X.XX.XXX.XX.XXXXXXX.XX.] duration:
5.001127496s trace:
https://jaeger.dev.storj.io/trace/4b593f15c1c6872a
FAIL (2025-12-29 13:10:16 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001177542s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [..........................X......................X....X.X...] duration:
13.297116744s trace:
https://jaeger.dev.storj.io/trace/141b6edfc3103b12
FAIL (2025-12-29 13:10:14 UTC)
orbiter warning: context deadline exceeded
Duration: 13.297141951s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
29 out of 60 is failed (threshold=3) [X...X..X....XXXX..XX..X...XX.X.XX...X.X.X.XXX.X.X.X.XX..X.XX] duration:
30.003446159s trace:
https://jaeger.dev.storj.io/trace/27989b20ffbf988b
FAIL (2025-12-29 13:10:01 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: RWV2X6L34VJAA2HLMXIOCFYR2LIK7QIG2Q47NFZ64QSE2V2MCJ6Q) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: JD2BE4P7CHPFKGXSV2BZPW6DLSBOYS52K3WQJ55QQLUZLMCB7FTA) context deadline exceeded
======LOGS======
Dec 29 13:09:31.700507957 (573h40m31.571590992s) [upload=502940]: Starting upload
Dec 29 13:09:31.700597066 (573h40m31.571680112s) [upload=502940]: Done starting upload
Dec 29 13:09:31.700672360 (573h40m31.571755405s) [upload=502940]: Uploading segments...
Dec 29 13:09:31.700799703 (573h40m31.571882738s) [upload=502940]: Got next segment. Inline: false
Dec 29 13:09:31.700846271 (573h40m31.571929306s) [upload=502940]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 13:09:31.700851953 (573h40m31.571934998s) [upload=502940]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 13:09:31.950417912 (573h40m31.821500967s) [seg_pos={0 0}, upload=502940]: Begin upload segment...
Dec 29 13:09:31.950474881 (573h40m31.821557926s) [seg_pos={0 0}, upload=502940]: Done begin upload segment.
Dec 29 13:09:31.950544594 (573h40m31.821627639s) [noise=true , address=145.79.227.39:4015 , piece_id=4RKI473ATNLSZSQPCSO3CMMVPYKYHZIFRPO25IG744KWM5GYSHBQ, seg_pos={0 0}, upload=502940]: Uploading piece...
Dec 29 13:09:31.950553180 (573h40m31.821636225s) [noise=true , address=145.79.227.26:4655 , piece_id=VDEVQ3UGLC4R5PJLUF4KBPRTWL7YJEOYCBJUUBQ7GR3CZHNEGO3Q, seg_pos={0 0}, upload=502940]: Uploading piece...
Dec 29 13:09:31.950594458 (573h40m31.821677503s) [noise=true , address=145.79.227.3:4151 , piece_id=EFZGJVWNUNRDZOXTDFWTWBASQLG2P3G2INK52ZO6NVBHO64K5L7A, seg_pos={0 0}, upload=502940]: Uploading piece...
Dec 29 13:09:31.951042071 (573h40m31.822125106s) [noise=true , address=145.79.227.6:4259 , piece_id=HAQJHHADLXYNAJYEYIA477JG2BPBMMGUIYDM6RFXWRADLJISDGHQ, seg_pos={0 0}, upload=502940]: Uploading piece...
Dec 29 13:09:31.951815834 (573h40m31.822898879s) [noise=true , address=77.90.31.11:4554 , piece_id=JKW3PBQW2XAETS5KRTWGZSVV7H5JK2PVM5FC7F4HCGKXFUXWL23Q, seg_pos={0 0}, upload=502940]: Uploading piece...
Dec 29 13:09:31.952025994 (573h40m31.823109039s) [noise=true , address=145.79.227.12:4767 , piece_id=ZS5HW3KPXNPLVJ2HNYXKXVRG74AJNAK6N6PCM6FN7FWZQ7P64GEA, seg_pos={0 0}, upload=502940]: Uploading piece...
Dec 29 13:09:31.952237927 (573h40m31.823320972s) [noise=true , address=45.91.116.14:4191 , piece_id=RWV2X6L34VJAA2HLMXIOCFYR2LIK7QIG2Q47NFZ64QSE2V2MCJ6Q, seg_pos={0 0}, upload=502940]: Uploading piece...
Dec 29 13:09:31.952832560 (573h40m31.823915605s) [noise=true , address=45.91.116.5:4297 , piece_id=4CSKRMO7VIVO5CMUCJO25QPCU73HH6J47FL4RLXKF6HMZGUFRTNA, seg_pos={0 0}, upload=502940]: Uploading piece...
Dec 29 13:09:31.953635458 (573h40m31.824718493s) [noise=true , address=145.79.227.17:4971 , piece_id=UNRPAUOPLSKA4QAODFLF3H2DY3U6WIPS266BBXSZJRKHMMQIPRXA, seg_pos={0 0}, upload=502940]: Uploading piece...
Dec 29 13:09:31.954715495 (573h40m31.82579854s) [noise=true , address=145.79.227.5:4221 , piece_id=MJHPEUTPNZ4KUTE5FKN6AIJK7DKFAAOOY7FODERK4BF7LLCN2HZA, seg_pos={0 0}, upload=502940]: Uploading piece...
Dec 29 13:09:31.955728223 (573h40m31.826811258s) [noise=true , address=145.79.227.28:4725 , piece_id=LFQQZY7PLVDIP5YTQH6P7WUCPRI4YJKXEUUV7HUONKWEJRWTHWLA, seg_pos={0 0}, upload=502940]: Uploading piece...
Dec 29 13:09:31.956287348 (573h40m31.827370393s) [noise=true , address=145.79.227.40:4051 , piece_id=FWF5DSHWX6CNNQKCIZ4YKJGHJTKWBYPR4LQEGYPI63P6H6Z4TAAQ, seg_pos={0 0}, upload=502940]: Uploading piece...
Dec 29 13:09:31.956306354 (573h40m31.827389389s) [noise=true , address=145.79.227.18:4006 , piece_id=HM47GSAZTBA6DKWGHSUNGQUFYSVTAERKJP3TKWKSB2J4MRKCICCA, seg_pos={0 0}, upload=502940]: Uploading piece...
Dec 29 13:09:31.957114713 (573h40m31.828197749s) [noise=true , address=45.91.116.6:4976 , piece_id=JD2BE4P7CHPFKGXSV2BZPW6DLSBOYS52K3WQJ55QQLUZLMCB7FTA, seg_pos={0 0}, upload=502940]: Uploading piece...
Dec 29 13:09:31.957650925 (573h40m31.82873397s) [noise=true , address=145.79.227.45:4230 , piece_id=LK6BJYK7JZ2SYESWGIDWZVQFHW3SLXE36PJLUW36ID22DE5SXOZQ, seg_pos={0 0}, upload=502940]: Uploading piece...
Dec 29 13:09:31.958682218 (573h40m31.829765253s) [noise=true , address=145.79.227.48:4302 , piece_id=KP2NBJQ6VRPOVO7TMM7EC2TLS6EDA3EBUNKBYSQB7VSBXXUI3AHQ, seg_pos={0 0}, upload=502940]: Uploading piece...
Dec 29 13:09:33.647677232 (573h40m33.518760277s) [upload=502940]: Next returned nil segment
Dec 29 13:09:33.647693773 (573h40m33.518776818s) [upload=502940]: Waiting for error group managing segments...
Dec 29 13:09:33.930296089 (573h40m33.801379124s) [noise=true , address=145.79.227.5:4221 , piece_id=MJHPEUTPNZ4KUTE5FKN6AIJK7DKFAAOOY7FODERK4BF7LLCN2HZA, seg_pos={0 0}, upload=502940]: Done uploading piece. err: <nil>
Dec 29 13:09:33.937491680 (573h40m33.808574725s) [noise=true , address=145.79.227.6:4259 , piece_id=HAQJHHADLXYNAJYEYIA477JG2BPBMMGUIYDM6RFXWRADLJISDGHQ, seg_pos={0 0}, upload=502940]: Done uploading piece. err: <nil>
Dec 29 13:09:33.938393197 (573h40m33.809476232s) [noise=true , address=145.79.227.26:4655 , piece_id=VDEVQ3UGLC4R5PJLUF4KBPRTWL7YJEOYCBJUUBQ7GR3CZHNEGO3Q, seg_pos={0 0}, upload=502940]: Done uploading piece. err: <nil>
Dec 29 13:09:33.941172720 (573h40m33.812255765s) [noise=true , address=145.79.227.3:4151 , piece_id=EFZGJVWNUNRDZOXTDFWTWBASQLG2P3G2INK52ZO6NVBHO64K5L7A, seg_pos={0 0}, upload=502940]: Done uploading piece. err: <nil>
Dec 29 13:09:33.941882080 (573h40m33.812965125s) [noise=true , address=145.79.227.17:4971 , piece_id=UNRPAUOPLSKA4QAODFLF3H2DY3U6WIPS266BBXSZJRKHMMQIPRXA, seg_pos={0 0}, upload=502940]: Done uploading piece. err: <nil>
Dec 29 13:09:33.942021937 (573h40m33.813104972s) [noise=true , address=145.79.227.39:4015 , piece_id=4RKI473ATNLSZSQPCSO3CMMVPYKYHZIFRPO25IG744KWM5GYSHBQ, seg_pos={0 0}, upload=502940]: Done uploading piece. err: <nil>
Dec 29 13:09:33.944489415 (573h40m33.81557246s) [noise=true , address=145.79.227.48:4302 , piece_id=KP2NBJQ6VRPOVO7TMM7EC2TLS6EDA3EBUNKBYSQB7VSBXXUI3AHQ, seg_pos={0 0}, upload=502940]: Done uploading piece. err: <nil>
Dec 29 13:09:33.947585932 (573h40m33.818668967s) [noise=true , address=145.79.227.28:4725 , piece_id=LFQQZY7PLVDIP5YTQH6P7WUCPRI4YJKXEUUV7HUONKWEJRWTHWLA, seg_pos={0 0}, upload=502940]: Done uploading piece. err: <nil>
Dec 29 13:09:33.954919075 (573h40m33.826002121s) [noise=true , address=145.79.227.12:4767 , piece_id=ZS5HW3KPXNPLVJ2HNYXKXVRG74AJNAK6N6PCM6FN7FWZQ7P64GEA, seg_pos={0 0}, upload=502940]: Done uploading piece. err: <nil>
Dec 29 13:09:33.958939340 (573h40m33.830022375s) [noise=true , address=145.79.227.45:4230 , piece_id=LK6BJYK7JZ2SYESWGIDWZVQFHW3SLXE36PJLUW36ID22DE5SXOZQ, seg_pos={0 0}, upload=502940]: Done uploading piece. err: <nil>
Dec 29 13:09:33.967314728 (573h40m33.838397763s) [noise=true , address=145.79.227.18:4006 , piece_id=HM47GSAZTBA6DKWGHSUNGQUFYSVTAERKJP3TKWKSB2J4MRKCICCA, seg_pos={0 0}, upload=502940]: Done uploading piece. err: <nil>
Dec 29 13:09:34.180692049 (573h40m34.051775095s) [noise=true , address=45.91.116.6:4976 , piece_id=JD2BE4P7CHPFKGXSV2BZPW6DLSBOYS52K3WQJ55QQLUZLMCB7FTA, seg_pos={0 0}, upload=502940]: Done uploading piece. err: <nil>
Dec 29 13:09:34.190213249 (573h40m34.061296284s) [noise=true , address=45.91.116.14:4191 , piece_id=RWV2X6L34VJAA2HLMXIOCFYR2LIK7QIG2Q47NFZ64QSE2V2MCJ6Q, seg_pos={0 0}, upload=502940]: Done uploading piece. err: <nil>
Dec 29 13:09:34.190263345 (573h40m34.06134639s) [seg_pos={0 0}, upload=502940]: Segment reached optimal threshold of 13 pieces.
Dec 29 13:09:34.190824634 (573h40m34.061907679s) [noise=true , address=77.90.31.11:4554 , piece_id=JKW3PBQW2XAETS5KRTWGZSVV7H5JK2PVM5FC7F4HCGKXFUXWL23Q, seg_pos={0 0}, upload=502940]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF
Dec 29 13:09:34.190835334 (573h40m34.061918379s) [noise=true , address=77.90.31.11:4554 , piece_id=JKW3PBQW2XAETS5KRTWGZSVV7H5JK2PVM5FC7F4HCGKXFUXWL23Q, seg_pos={0 0}, upload=502940]: Long tail cancellation due to optimal threshold
Dec 29 13:09:34.191155223 (573h40m34.062241374s) [noise=true , address=145.79.227.40:4051 , piece_id=FWF5DSHWX6CNNQKCIZ4YKJGHJTKWBYPR4LQEGYPI63P6H6Z4TAAQ, seg_pos={0 0}, upload=502940]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 29 13:09:34.191196711 (573h40m34.062279746s) [noise=true , address=145.79.227.40:4051 , piece_id=FWF5DSHWX6CNNQKCIZ4YKJGHJTKWBYPR4LQEGYPI63P6H6Z4TAAQ, seg_pos={0 0}, upload=502940]: Long tail cancellation due to optimal threshold
Dec 29 13:09:34.191773680 (573h40m34.062856726s) [noise=true , address=45.91.116.5:4297 , piece_id=4CSKRMO7VIVO5CMUCJO25QPCU73HH6J47FL4RLXKF6HMZGUFRTNA, seg_pos={0 0}, upload=502940]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF
Dec 29 13:09:34.191791976 (573h40m34.062875011s) [noise=true , address=45.91.116.5:4297 , piece_id=4CSKRMO7VIVO5CMUCJO25QPCU73HH6J47FL4RLXKF6HMZGUFRTNA, seg_pos={0 0}, upload=502940]: Long tail cancellation due to optimal threshold
Dec 29 13:09:34.192027333 (573h40m34.063110368s) [seg_pos={0 0}, upload=502940]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 13:09:34.192390796 (573h40m34.063473841s) [upload=502940]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:09:34.192425522 (573h40m34.063508577s) [upload=502940]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:09:34.479795967 (573h40m34.350879012s) [upload=502940]: Done uploading segments...
Dec 29 13:09:34.479819722 (573h40m34.350902767s) [upload=502940]: Upload finished. err: <nil>
Duration: 30.003617646s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [...........................X......................X....X.X..] duration:
18.250682856s trace:
https://jaeger.dev.storj.io/trace/141b6edfc3103b12
FAIL (2025-12-29 13:09:56 UTC)
orbiter warning: context deadline exceeded
Duration: 18.250722501s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [..........X......................X.....X....................] duration:
39.200388729s trace:
https://jaeger.dev.storj.io/trace/213826fef0d4f1dd
FAIL (2025-12-29 13:09:50 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
c 29 12:52:02.421331209 (573h23m2.292414254s) [noise=true , address=145.79.227.48:4334 , piece_id=MFEYVH6L2DBOS7Z444QFOQDVQCQS5MG3M32335L7ZCN4MXVH5L2A, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.421480444 (573h23m2.292563479s) [noise=true , address=77.90.31.17:4799 , piece_id=DFRKPKCVCIPZZJB6QDU64E5JGJIIKXDEO5MJAA4X5V6N3ZTEGTFA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.422126223 (573h23m2.293209268s) [noise=true , address=77.90.31.14:4676 , piece_id=RGKFUGBFBMHXICXNPPCFXNFHOZ5EXIFPT54PBT2EA5AS2ZC5OTGA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.422527958 (573h23m2.293611013s) [noise=true , address=145.79.227.17:4980 , piece_id=D3CAP7TOKGQ2NVULNTN3WSB2OQKOLVWN7TFGLPOLLJPPMHHZR7NQ, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.422668796 (573h23m2.293751841s) [noise=true , address=193.149.164.143:4697 , piece_id=C2JF73NKW64HNXGS5BJTVZRCAIVWKZBGDGUZFJMDUWW3RBBXJDSA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.422966322 (573h23m2.294049357s) [noise=true , address=77.90.31.19:4863 , piece_id=XLK3SHF2GNV625H7RC5SGL67TKKBAJREGPTBCEJDX3QGD6H63GMA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.423193946 (573h23m2.294276981s) [noise=true , address=77.90.31.18:4829 , piece_id=RG37QNRSJFFED3XOPAEOHGJNJ45MEMUON3R3MB5ASEXS4A4EQ42Q, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.423513825 (573h23m2.29459687s) [noise=true , address=145.79.227.43:4166 , piece_id=QPEP6OT7WAFMRNNI6XVL25H3FAJ76FCQXVSEACOTV6XCYIZZSVPA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.423810040 (573h23m2.294893085s) [noise=true , address=77.90.31.10:4545 , piece_id=NE3YEJJJKKOQSM2JKO5PFTPJH7SG65UZGR3QJP5JZXOUF3FN32YQ, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.424441803 (573h23m2.295524868s) [noise=true , address=64.49.13.169:4708 , piece_id=5ORIIBIW2GPJRDD5C7IBQDPKPS5SJBTSVR6SGC6NKX36DTTGAYGA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.425051363 (573h23m2.296134408s) [noise=true , address=45.91.116.5:4323 , piece_id=7FAJKQP23A6U6SXPWVVAFVIHQA54JSPERWMCSONK5NYNHXRDLF3A, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.425681483 (573h23m2.296764528s) [noise=true , address=145.79.227.7:4303 , piece_id=MWIQ7KUZ534TFKQFMQADMBPKUFDGNFFK3JJEH3KF2CT6IQGHQRKQ, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.426480093 (573h23m2.297563138s) [noise=true , address=145.79.227.33:4877 , piece_id=IFC4ANBQU7ORCMSTYAAG3QKE4S2JPCWKDD47P5Z57V6TWAR6GFTA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.427077340 (573h23m2.298160385s) [noise=true , address=145.79.227.34:4937 , piece_id=2EYFRE3YDBIZK7YORZMWRETOEHJGDDKJVZBFW36FU6LX6PHRTDIA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.427529692 (573h23m2.298612737s) [noise=true , address=145.79.227.30:4793 , piece_id=AGUHSTJNNNRAAP5TKWIONYBZDKPJDJEQX7RX5GEGWN6TG4RELAMQ, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.428130125 (573h23m2.29921317s) [noise=true , address=77.90.31.16:4747 , piece_id=5TQY6C4SF6JQT6KTSEMOSY65TLPLUVCDTNVJ6U2QYFMTYP2G73VA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:03.783583605 (573h23m3.65466665s) [upload=502680]: Next returned nil segment
Dec 29 12:52:03.783597080 (573h23m3.654680125s) [upload=502680]: Waiting for error group managing segments...
Dec 29 12:52:04.132513046 (573h23m4.003596091s) [noise=true , address=64.49.13.137:4494 , piece_id=K4TIPJCGNFYIMSQKXQAWABJLIHHTAS3F36I723BHTJ7YG4HUKIIQ, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.271420967 (573h23m4.142504012s) [noise=true , address=145.79.227.7:4303 , piece_id=MWIQ7KUZ534TFKQFMQADMBPKUFDGNFFK3JJEH3KF2CT6IQGHQRKQ, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.275668766 (573h23m4.146751811s) [noise=true , address=145.79.227.48:4334 , piece_id=MFEYVH6L2DBOS7Z444QFOQDVQCQS5MG3M32335L7ZCN4MXVH5L2A, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.298256927 (573h23m4.169339972s) [noise=true , address=145.79.227.17:4980 , piece_id=D3CAP7TOKGQ2NVULNTN3WSB2OQKOLVWN7TFGLPOLLJPPMHHZR7NQ, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.487163872 (573h23m4.358246917s) [noise=true , address=77.90.31.19:4863 , piece_id=XLK3SHF2GNV625H7RC5SGL67TKKBAJREGPTBCEJDX3QGD6H63GMA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.563458298 (573h23m4.434541343s) [noise=true , address=45.91.116.5:4323 , piece_id=7FAJKQP23A6U6SXPWVVAFVIHQA54JSPERWMCSONK5NYNHXRDLF3A, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.572838760 (573h23m4.443921795s) [noise=true , address=77.90.31.10:4545 , piece_id=NE3YEJJJKKOQSM2JKO5PFTPJH7SG65UZGR3QJP5JZXOUF3FN32YQ, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.580866937 (573h23m4.451949972s) [noise=true , address=77.90.31.14:4676 , piece_id=RGKFUGBFBMHXICXNPPCFXNFHOZ5EXIFPT54PBT2EA5AS2ZC5OTGA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.595018524 (573h23m4.466101569s) [noise=true , address=77.90.31.17:4799 , piece_id=DFRKPKCVCIPZZJB6QDU64E5JGJIIKXDEO5MJAA4X5V6N3ZTEGTFA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:05.183831178 (573h23m5.054914213s) [noise=true , address=145.79.227.30:4793 , piece_id=AGUHSTJNNNRAAP5TKWIONYBZDKPJDJEQX7RX5GEGWN6TG4RELAMQ, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:05.242293548 (573h23m5.113376593s) [noise=true , address=145.79.227.33:4877 , piece_id=IFC4ANBQU7ORCMSTYAAG3QKE4S2JPCWKDD47P5Z57V6TWAR6GFTA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:05.255836156 (573h23m5.126919201s) [noise=true , address=145.79.227.34:4937 , piece_id=2EYFRE3YDBIZK7YORZMWRETOEHJGDDKJVZBFW36FU6LX6PHRTDIA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:05.305203848 (573h23m5.176286883s) [noise=true , address=145.79.227.40:4058 , piece_id=EN2T5WMAEV3PVNAOWLMN4BUIJSAXRMAIQBQ6PKQ7PWPATZLYYBIQ, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:05.326720429 (573h23m5.197803474s) [noise=true , address=145.79.227.43:4166 , piece_id=QPEP6OT7WAFMRNNI6XVL25H3FAJ76FCQXVSEACOTV6XCYIZZSVPA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:05.326747741 (573h23m5.197830786s) [seg_pos={0 0}, upload=502680]: Segment reached optimal threshold of 14 pieces.
Dec 29 12:52:05.327021833 (573h23m5.198104889s) [noise=true , address=77.90.31.18:4829 , piece_id=RG37QNRSJFFED3XOPAEOHGJNJ45MEMUON3R3MB5ASEXS4A4EQ42Q, seg_pos={0 0}, upload=502680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo): protocol: expected piece hash; context canceled; EOF
Dec 29 12:52:05.327031712 (573h23m5.198114757s) [noise=true , address=77.90.31.18:4829 , piece_id=RG37QNRSJFFED3XOPAEOHGJNJ45MEMUON3R3MB5ASEXS4A4EQ42Q, seg_pos={0 0}, upload=502680]: Long tail cancellation due to optimal threshold
Dec 29 12:52:05.327058794 (573h23m5.198141849s) [noise=true , address=64.49.13.169:4708 , piece_id=5ORIIBIW2GPJRDD5C7IBQDPKPS5SJBTSVR6SGC6NKX36DTTGAYGA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:40464->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:40464->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40464->64.49.13.169:4708: use of closed network connection
Dec 29 12:52:05.327066769 (573h23m5.198149814s) [noise=true , address=64.49.13.169:4708 , piece_id=5ORIIBIW2GPJRDD5C7IBQDPKPS5SJBTSVR6SGC6NKX36DTTGAYGA, seg_pos={0 0}, upload=502680]: Long tail cancellation due to optimal threshold
Dec 29 12:52:05.327203149 (573h23m5.198286204s) [noise=true , address=77.90.31.16:4747 , piece_id=5TQY6C4SF6JQT6KTSEMOSY65TLPLUVCDTNVJ6U2QYFMTYP2G73VA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2): protocol: expected piece hash; context canceled; EOF
Dec 29 12:52:05.327213137 (573h23m5.198296182s) [noise=true , address=77.90.31.16:4747 , piece_id=5TQY6C4SF6JQT6KTSEMOSY65TLPLUVCDTNVJ6U2QYFMTYP2G73VA, seg_pos={0 0}, upload=502680]: Long tail cancellation due to optimal threshold
Dec 29 12:52:05.327491227 (573h23m5.198574282s) [noise=true , address=193.149.164.143:4697 , piece_id=C2JF73NKW64HNXGS5BJTVZRCAIVWKZBGDGUZFJMDUWW3RBBXJDSA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF
Dec 29 12:52:05.327500265 (573h23m5.1985833s) [noise=true , address=193.149.164.143:4697 , piece_id=C2JF73NKW64HNXGS5BJTVZRCAIVWKZBGDGUZFJMDUWW3RBBXJDSA, seg_pos={0 0}, upload=502680]: Long tail cancellation due to optimal threshold
Dec 29 12:52:05.327548276 (573h23m5.198631321s) [seg_pos={0 0}, upload=502680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 12:52:05.327736143 (573h23m5.198819188s) [upload=502680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 12:52:05.327744139 (573h23m5.198827174s) [upload=502680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 12:52:05.624748327 (573h23m5.495831362s) [upload=502680]: Done uploading segments...
Dec 29 12:52:05.624772935 (573h23m5.49585597s) [upload=502680]: Upload finished. err: <nil>
Duration: 39.200403878s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
52 out of 60 is failed (threshold=6) [XXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXX.X.XX.XXX.XX.XXXXXXX.XX] duration:
5.001137795s trace:
https://jaeger.dev.storj.io/trace/4b593f15c1c6872a
FAIL (2025-12-29 13:09:26 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001153867s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [............................X......................X....X.X.] duration:
19.88318276s trace:
https://jaeger.dev.storj.io/trace/141b6edfc3103b12
FAIL (2025-12-29 13:09:24 UTC)
orbiter warning: context deadline exceeded
Duration: 19.883196976s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
28 out of 60 is failed (threshold=3) [.X...X..X....XXXX..XX..X...XX.X.XX...X.X.X.XXX.X.X.X.XX..X.X] duration:
30.00368661s trace:
https://jaeger.dev.storj.io/trace/cbdd80ef5cd0e95
FAIL (2025-12-29 13:09:17 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: QTGYJ4IEJN254VKL66ZYDP53ZU47U5QEUKL555OSVLTQ5UDFCI4A) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: 7GWJ3BBQF4PWR7E7IDGNOFUGJ62JDSD6UCYYQSPRDZCGQZ7Y767A) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: CPT3QA5JTTISDATLC2DBA2TSQOK7PSZ2DI4FUPHJ5WR6XCCQITBQ) context deadline exceeded
======LOGS======
Dec 29 13:08:48.639119598 (573h39m48.510202633s) [upload=502930]: Starting upload
Dec 29 13:08:48.639175414 (573h39m48.510258449s) [upload=502930]: Done starting upload
Dec 29 13:08:48.639313467 (573h39m48.510396502s) [upload=502930]: Uploading segments...
Dec 29 13:08:48.639390875 (573h39m48.51047391s) [upload=502930]: Got next segment. Inline: false
Dec 29 13:08:48.639418747 (573h39m48.510501782s) [upload=502930]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 13:08:48.639423316 (573h39m48.510506351s) [upload=502930]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 13:08:48.887742773 (573h39m48.758825808s) [seg_pos={0 0}, upload=502930]: Begin upload segment...
Dec 29 13:08:48.887801294 (573h39m48.758884329s) [seg_pos={0 0}, upload=502930]: Done begin upload segment.
Dec 29 13:08:48.887853393 (573h39m48.758936428s) [noise=true , address=45.91.116.9:4261 , piece_id=SHIJHYSAQAO4IOIXRZN3QVSOKU6BLNXI72NXBXCYVVNWQCOFR6TQ, seg_pos={0 0}, upload=502930]: Uploading piece...
Dec 29 13:08:48.887869284 (573h39m48.758952319s) [noise=true , address=45.91.116.14:4192 , piece_id=7GWJ3BBQF4PWR7E7IDGNOFUGJ62JDSD6UCYYQSPRDZCGQZ7Y767A, seg_pos={0 0}, upload=502930]: Uploading piece...
Dec 29 13:08:48.888569487 (573h39m48.759652522s) [noise=true , address=145.79.227.18:4007 , piece_id=EDYHSGOSUEGEKWZIRIYSSFIR5IF6Z2YBITQQ3Z3WKIMTRWHRTPAA, seg_pos={0 0}, upload=502930]: Uploading piece...
Dec 29 13:08:48.888920325 (573h39m48.76000336s) [noise=true , address=145.79.227.15:4875 , piece_id=ZZK3WP4LJKNVUIKC2OISEAD5X7ZMO3JPELXKCKUH3A5LDX6M7ZAQ, seg_pos={0 0}, upload=502930]: Uploading piece...
Dec 29 13:08:48.889236748 (573h39m48.760319783s) [noise=true , address=145.79.227.7:4294 , piece_id=INIDWEU3E6VWPL2W2WC2TRDZ5EMDD4XTFUCVP7I6LLOR4KCUJVKA, seg_pos={0 0}, upload=502930]: Uploading piece...
Dec 29 13:08:48.889589890 (573h39m48.760672925s) [noise=true , address=77.90.31.13:4626 , piece_id=LTG626FTTMVYNVZUWXDMQDODVR357KLPEPAP6LW7EUR4D2JPPLMA, seg_pos={0 0}, upload=502930]: Uploading piece...
Dec 29 13:08:48.889923124 (573h39m48.761006159s) [noise=true , address=145.79.227.42:4122 , piece_id=NADEABRC5SP33F4SBQJZHJOSVNEK56SZF2HUKG7TDYHHC656SV4Q, seg_pos={0 0}, upload=502930]: Uploading piece...
Dec 29 13:08:48.890362982 (573h39m48.761446027s) [noise=true , address=145.79.227.19:4077 , piece_id=TCU7H3D2O7O2ATNVDNIKSSBOD6ZUBAOAF5CJAX34ZBVVF5G4G6BQ, seg_pos={0 0}, upload=502930]: Uploading piece...
Dec 29 13:08:48.890704222 (573h39m48.761787257s) [noise=true , address=77.90.31.14:4664 , piece_id=CPT3QA5JTTISDATLC2DBA2TSQOK7PSZ2DI4FUPHJ5WR6XCCQITBQ, seg_pos={0 0}, upload=502930]: Uploading piece...
Dec 29 13:08:48.891018119 (573h39m48.762101154s) [noise=true , address=145.79.227.11:4730 , piece_id=TRP6UCJM6LV2CG2ZKNR3SF35RAWNGH56XCJ24OZ7DSBKDO2CSMPQ, seg_pos={0 0}, upload=502930]: Uploading piece...
Dec 29 13:08:48.891045512 (573h39m48.762128547s) [noise=true , address=145.79.227.49:4337 , piece_id=3EGZXBYIX3VFBKDNL7CWHP6MPKRYE6MJZ6MOOJGQ4KLK5PPOXNFA, seg_pos={0 0}, upload=502930]: Uploading piece...
Dec 29 13:08:48.891317480 (573h39m48.762400515s) [noise=true , address=45.91.116.6:4976 , piece_id=RKAXNG5TA2HKD5SIUORTAMPKBBWLMUCVCDRPAPM4CZ5263IY3IWQ, seg_pos={0 0}, upload=502930]: Uploading piece...
Dec 29 13:08:48.891555352 (573h39m48.762638387s) [noise=true , address=77.90.31.10:4518 , piece_id=P3SKO7TDGVHQMWKP4A6STRIMMGLJYONGQIDX3HQF7WJUGP22ZVMQ, seg_pos={0 0}, upload=502930]: Uploading piece...
Dec 29 13:08:48.891624113 (573h39m48.762707148s) [noise=true , address=145.79.227.35:4483 , piece_id=EBPENH3U3NQPEIA4H2HFIC7LU5AOGPP5XE5VJCBM376B3DWURR3A, seg_pos={0 0}, upload=502930]: Uploading piece...
Dec 29 13:08:48.891909497 (573h39m48.762992542s) [noise=true , address=145.79.227.37:4566 , piece_id=GOEPNYQICXCMJXBG3ZJIFY5GDW7RBMYVJSOJJBF2FSRV6HRZKTBQ, seg_pos={0 0}, upload=502930]: Uploading piece...
Dec 29 13:08:48.892270113 (573h39m48.763353148s) [noise=true , address=45.91.116.13:4154 , piece_id=QTGYJ4IEJN254VKL66ZYDP53ZU47U5QEUKL555OSVLTQ5UDFCI4A, seg_pos={0 0}, upload=502930]: Uploading piece...
Dec 29 13:08:50.599546487 (573h39m50.470629532s) [upload=502930]: Next returned nil segment
Dec 29 13:08:50.599564180 (573h39m50.470647225s) [upload=502930]: Waiting for error group managing segments...
Dec 29 13:08:50.876589988 (573h39m50.747673023s) [noise=true , address=145.79.227.18:4007 , piece_id=EDYHSGOSUEGEKWZIRIYSSFIR5IF6Z2YBITQQ3Z3WKIMTRWHRTPAA, seg_pos={0 0}, upload=502930]: Done uploading piece. err: <nil>
Dec 29 13:08:50.877199147 (573h39m50.748282193s) [noise=true , address=145.79.227.15:4875 , piece_id=ZZK3WP4LJKNVUIKC2OISEAD5X7ZMO3JPELXKCKUH3A5LDX6M7ZAQ, seg_pos={0 0}, upload=502930]: Done uploading piece. err: <nil>
Dec 29 13:08:50.881004664 (573h39m50.752087709s) [noise=true , address=145.79.227.19:4077 , piece_id=TCU7H3D2O7O2ATNVDNIKSSBOD6ZUBAOAF5CJAX34ZBVVF5G4G6BQ, seg_pos={0 0}, upload=502930]: Done uploading piece. err: <nil>
Dec 29 13:08:50.885176377 (573h39m50.756259412s) [noise=true , address=145.79.227.42:4122 , piece_id=NADEABRC5SP33F4SBQJZHJOSVNEK56SZF2HUKG7TDYHHC656SV4Q, seg_pos={0 0}, upload=502930]: Done uploading piece. err: <nil>
Dec 29 13:08:50.894021700 (573h39m50.765104735s) [noise=true , address=145.79.227.37:4566 , piece_id=GOEPNYQICXCMJXBG3ZJIFY5GDW7RBMYVJSOJJBF2FSRV6HRZKTBQ, seg_pos={0 0}, upload=502930]: Done uploading piece. err: <nil>
Dec 29 13:08:50.895881551 (573h39m50.766964596s) [noise=true , address=145.79.227.11:4730 , piece_id=TRP6UCJM6LV2CG2ZKNR3SF35RAWNGH56XCJ24OZ7DSBKDO2CSMPQ, seg_pos={0 0}, upload=502930]: Done uploading piece. err: <nil>
Dec 29 13:08:50.902223978 (573h39m50.773307013s) [noise=true , address=145.79.227.7:4294 , piece_id=INIDWEU3E6VWPL2W2WC2TRDZ5EMDD4XTFUCVP7I6LLOR4KCUJVKA, seg_pos={0 0}, upload=502930]: Done uploading piece. err: <nil>
Dec 29 13:08:51.087995545 (573h39m50.95907859s) [noise=true , address=77.90.31.10:4518 , piece_id=P3SKO7TDGVHQMWKP4A6STRIMMGLJYONGQIDX3HQF7WJUGP22ZVMQ, seg_pos={0 0}, upload=502930]: Done uploading piece. err: <nil>
Dec 29 13:08:51.174174212 (573h39m51.045257247s) [noise=true , address=45.91.116.6:4976 , piece_id=RKAXNG5TA2HKD5SIUORTAMPKBBWLMUCVCDRPAPM4CZ5263IY3IWQ, seg_pos={0 0}, upload=502930]: Done uploading piece. err: <nil>
Dec 29 13:08:51.232376047 (573h39m51.103459082s) [noise=true , address=45.91.116.9:4261 , piece_id=SHIJHYSAQAO4IOIXRZN3QVSOKU6BLNXI72NXBXCYVVNWQCOFR6TQ, seg_pos={0 0}, upload=502930]: Done uploading piece. err: <nil>
Dec 29 13:08:51.244642403 (573h39m51.115725438s) [noise=true , address=45.91.116.14:4192 , piece_id=7GWJ3BBQF4PWR7E7IDGNOFUGJ62JDSD6UCYYQSPRDZCGQZ7Y767A, seg_pos={0 0}, upload=502930]: Done uploading piece. err: <nil>
Dec 29 13:08:51.717175526 (573h39m51.588258571s) [noise=true , address=77.90.31.14:4664 , piece_id=CPT3QA5JTTISDATLC2DBA2TSQOK7PSZ2DI4FUPHJ5WR6XCCQITBQ, seg_pos={0 0}, upload=502930]: Done uploading piece. err: <nil>
Dec 29 13:08:51.818365596 (573h39m51.689448631s) [noise=true , address=45.91.116.13:4154 , piece_id=QTGYJ4IEJN254VKL66ZYDP53ZU47U5QEUKL555OSVLTQ5UDFCI4A, seg_pos={0 0}, upload=502930]: Done uploading piece. err: <nil>
Dec 29 13:08:51.818384793 (573h39m51.689467828s) [seg_pos={0 0}, upload=502930]: Segment reached optimal threshold of 13 pieces.
Dec 29 13:08:51.818557352 (573h39m51.689640387s) [noise=true , address=145.79.227.35:4483 , piece_id=EBPENH3U3NQPEIA4H2HFIC7LU5AOGPP5XE5VJCBM376B3DWURR3A, seg_pos={0 0}, upload=502930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF
Dec 29 13:08:51.818563764 (573h39m51.689646799s) [noise=true , address=145.79.227.35:4483 , piece_id=EBPENH3U3NQPEIA4H2HFIC7LU5AOGPP5XE5VJCBM376B3DWURR3A, seg_pos={0 0}, upload=502930]: Long tail cancellation due to optimal threshold
Dec 29 13:08:51.818726945 (573h39m51.68980998s) [noise=true , address=145.79.227.49:4337 , piece_id=3EGZXBYIX3VFBKDNL7CWHP6MPKRYE6MJZ6MOOJGQ4KLK5PPOXNFA, seg_pos={0 0}, upload=502930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: expected piece hash; context canceled; EOF
Dec 29 13:08:51.818732716 (573h39m51.689815751s) [noise=true , address=145.79.227.49:4337 , piece_id=3EGZXBYIX3VFBKDNL7CWHP6MPKRYE6MJZ6MOOJGQ4KLK5PPOXNFA, seg_pos={0 0}, upload=502930]: Long tail cancellation due to optimal threshold
Dec 29 13:08:51.818738016 (573h39m51.689821051s) [noise=true , address=77.90.31.13:4626 , piece_id=LTG626FTTMVYNVZUWXDMQDODVR357KLPEPAP6LW7EUR4D2JPPLMA, seg_pos={0 0}, upload=502930]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: read tcp 188.166.225.170:52206->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 29 13:08:51.818746101 (573h39m51.689829136s) [noise=true , address=77.90.31.13:4626 , piece_id=LTG626FTTMVYNVZUWXDMQDODVR357KLPEPAP6LW7EUR4D2JPPLMA, seg_pos={0 0}, upload=502930]: Long tail cancellation due to optimal threshold
Dec 29 13:08:51.818771710 (573h39m51.689854745s) [seg_pos={0 0}, upload=502930]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 13:08:51.818832987 (573h39m51.689916022s) [upload=502930]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:08:51.818838096 (573h39m51.689921131s) [upload=502930]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:08:52.119439023 (573h39m51.990522078s) [upload=502930]: Done uploading segments...
Dec 29 13:08:52.119505740 (573h39m51.990588775s) [upload=502930]: Upload finished. err: <nil>
Duration: 30.003707189s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [X..........X......................X.....X...................] duration:
47.139935573s trace:
https://jaeger.dev.storj.io/trace/213826fef0d4f1dd
FAIL (2025-12-29 13:09:11 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
c 29 12:52:02.421331209 (573h23m2.292414254s) [noise=true , address=145.79.227.48:4334 , piece_id=MFEYVH6L2DBOS7Z444QFOQDVQCQS5MG3M32335L7ZCN4MXVH5L2A, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.421480444 (573h23m2.292563479s) [noise=true , address=77.90.31.17:4799 , piece_id=DFRKPKCVCIPZZJB6QDU64E5JGJIIKXDEO5MJAA4X5V6N3ZTEGTFA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.422126223 (573h23m2.293209268s) [noise=true , address=77.90.31.14:4676 , piece_id=RGKFUGBFBMHXICXNPPCFXNFHOZ5EXIFPT54PBT2EA5AS2ZC5OTGA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.422527958 (573h23m2.293611013s) [noise=true , address=145.79.227.17:4980 , piece_id=D3CAP7TOKGQ2NVULNTN3WSB2OQKOLVWN7TFGLPOLLJPPMHHZR7NQ, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.422668796 (573h23m2.293751841s) [noise=true , address=193.149.164.143:4697 , piece_id=C2JF73NKW64HNXGS5BJTVZRCAIVWKZBGDGUZFJMDUWW3RBBXJDSA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.422966322 (573h23m2.294049357s) [noise=true , address=77.90.31.19:4863 , piece_id=XLK3SHF2GNV625H7RC5SGL67TKKBAJREGPTBCEJDX3QGD6H63GMA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.423193946 (573h23m2.294276981s) [noise=true , address=77.90.31.18:4829 , piece_id=RG37QNRSJFFED3XOPAEOHGJNJ45MEMUON3R3MB5ASEXS4A4EQ42Q, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.423513825 (573h23m2.29459687s) [noise=true , address=145.79.227.43:4166 , piece_id=QPEP6OT7WAFMRNNI6XVL25H3FAJ76FCQXVSEACOTV6XCYIZZSVPA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.423810040 (573h23m2.294893085s) [noise=true , address=77.90.31.10:4545 , piece_id=NE3YEJJJKKOQSM2JKO5PFTPJH7SG65UZGR3QJP5JZXOUF3FN32YQ, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.424441803 (573h23m2.295524868s) [noise=true , address=64.49.13.169:4708 , piece_id=5ORIIBIW2GPJRDD5C7IBQDPKPS5SJBTSVR6SGC6NKX36DTTGAYGA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.425051363 (573h23m2.296134408s) [noise=true , address=45.91.116.5:4323 , piece_id=7FAJKQP23A6U6SXPWVVAFVIHQA54JSPERWMCSONK5NYNHXRDLF3A, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.425681483 (573h23m2.296764528s) [noise=true , address=145.79.227.7:4303 , piece_id=MWIQ7KUZ534TFKQFMQADMBPKUFDGNFFK3JJEH3KF2CT6IQGHQRKQ, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.426480093 (573h23m2.297563138s) [noise=true , address=145.79.227.33:4877 , piece_id=IFC4ANBQU7ORCMSTYAAG3QKE4S2JPCWKDD47P5Z57V6TWAR6GFTA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.427077340 (573h23m2.298160385s) [noise=true , address=145.79.227.34:4937 , piece_id=2EYFRE3YDBIZK7YORZMWRETOEHJGDDKJVZBFW36FU6LX6PHRTDIA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.427529692 (573h23m2.298612737s) [noise=true , address=145.79.227.30:4793 , piece_id=AGUHSTJNNNRAAP5TKWIONYBZDKPJDJEQX7RX5GEGWN6TG4RELAMQ, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.428130125 (573h23m2.29921317s) [noise=true , address=77.90.31.16:4747 , piece_id=5TQY6C4SF6JQT6KTSEMOSY65TLPLUVCDTNVJ6U2QYFMTYP2G73VA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:03.783583605 (573h23m3.65466665s) [upload=502680]: Next returned nil segment
Dec 29 12:52:03.783597080 (573h23m3.654680125s) [upload=502680]: Waiting for error group managing segments...
Dec 29 12:52:04.132513046 (573h23m4.003596091s) [noise=true , address=64.49.13.137:4494 , piece_id=K4TIPJCGNFYIMSQKXQAWABJLIHHTAS3F36I723BHTJ7YG4HUKIIQ, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.271420967 (573h23m4.142504012s) [noise=true , address=145.79.227.7:4303 , piece_id=MWIQ7KUZ534TFKQFMQADMBPKUFDGNFFK3JJEH3KF2CT6IQGHQRKQ, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.275668766 (573h23m4.146751811s) [noise=true , address=145.79.227.48:4334 , piece_id=MFEYVH6L2DBOS7Z444QFOQDVQCQS5MG3M32335L7ZCN4MXVH5L2A, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.298256927 (573h23m4.169339972s) [noise=true , address=145.79.227.17:4980 , piece_id=D3CAP7TOKGQ2NVULNTN3WSB2OQKOLVWN7TFGLPOLLJPPMHHZR7NQ, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.487163872 (573h23m4.358246917s) [noise=true , address=77.90.31.19:4863 , piece_id=XLK3SHF2GNV625H7RC5SGL67TKKBAJREGPTBCEJDX3QGD6H63GMA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.563458298 (573h23m4.434541343s) [noise=true , address=45.91.116.5:4323 , piece_id=7FAJKQP23A6U6SXPWVVAFVIHQA54JSPERWMCSONK5NYNHXRDLF3A, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.572838760 (573h23m4.443921795s) [noise=true , address=77.90.31.10:4545 , piece_id=NE3YEJJJKKOQSM2JKO5PFTPJH7SG65UZGR3QJP5JZXOUF3FN32YQ, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.580866937 (573h23m4.451949972s) [noise=true , address=77.90.31.14:4676 , piece_id=RGKFUGBFBMHXICXNPPCFXNFHOZ5EXIFPT54PBT2EA5AS2ZC5OTGA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.595018524 (573h23m4.466101569s) [noise=true , address=77.90.31.17:4799 , piece_id=DFRKPKCVCIPZZJB6QDU64E5JGJIIKXDEO5MJAA4X5V6N3ZTEGTFA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:05.183831178 (573h23m5.054914213s) [noise=true , address=145.79.227.30:4793 , piece_id=AGUHSTJNNNRAAP5TKWIONYBZDKPJDJEQX7RX5GEGWN6TG4RELAMQ, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:05.242293548 (573h23m5.113376593s) [noise=true , address=145.79.227.33:4877 , piece_id=IFC4ANBQU7ORCMSTYAAG3QKE4S2JPCWKDD47P5Z57V6TWAR6GFTA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:05.255836156 (573h23m5.126919201s) [noise=true , address=145.79.227.34:4937 , piece_id=2EYFRE3YDBIZK7YORZMWRETOEHJGDDKJVZBFW36FU6LX6PHRTDIA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:05.305203848 (573h23m5.176286883s) [noise=true , address=145.79.227.40:4058 , piece_id=EN2T5WMAEV3PVNAOWLMN4BUIJSAXRMAIQBQ6PKQ7PWPATZLYYBIQ, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:05.326720429 (573h23m5.197803474s) [noise=true , address=145.79.227.43:4166 , piece_id=QPEP6OT7WAFMRNNI6XVL25H3FAJ76FCQXVSEACOTV6XCYIZZSVPA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:05.326747741 (573h23m5.197830786s) [seg_pos={0 0}, upload=502680]: Segment reached optimal threshold of 14 pieces.
Dec 29 12:52:05.327021833 (573h23m5.198104889s) [noise=true , address=77.90.31.18:4829 , piece_id=RG37QNRSJFFED3XOPAEOHGJNJ45MEMUON3R3MB5ASEXS4A4EQ42Q, seg_pos={0 0}, upload=502680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo): protocol: expected piece hash; context canceled; EOF
Dec 29 12:52:05.327031712 (573h23m5.198114757s) [noise=true , address=77.90.31.18:4829 , piece_id=RG37QNRSJFFED3XOPAEOHGJNJ45MEMUON3R3MB5ASEXS4A4EQ42Q, seg_pos={0 0}, upload=502680]: Long tail cancellation due to optimal threshold
Dec 29 12:52:05.327058794 (573h23m5.198141849s) [noise=true , address=64.49.13.169:4708 , piece_id=5ORIIBIW2GPJRDD5C7IBQDPKPS5SJBTSVR6SGC6NKX36DTTGAYGA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:40464->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:40464->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40464->64.49.13.169:4708: use of closed network connection
Dec 29 12:52:05.327066769 (573h23m5.198149814s) [noise=true , address=64.49.13.169:4708 , piece_id=5ORIIBIW2GPJRDD5C7IBQDPKPS5SJBTSVR6SGC6NKX36DTTGAYGA, seg_pos={0 0}, upload=502680]: Long tail cancellation due to optimal threshold
Dec 29 12:52:05.327203149 (573h23m5.198286204s) [noise=true , address=77.90.31.16:4747 , piece_id=5TQY6C4SF6JQT6KTSEMOSY65TLPLUVCDTNVJ6U2QYFMTYP2G73VA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2): protocol: expected piece hash; context canceled; EOF
Dec 29 12:52:05.327213137 (573h23m5.198296182s) [noise=true , address=77.90.31.16:4747 , piece_id=5TQY6C4SF6JQT6KTSEMOSY65TLPLUVCDTNVJ6U2QYFMTYP2G73VA, seg_pos={0 0}, upload=502680]: Long tail cancellation due to optimal threshold
Dec 29 12:52:05.327491227 (573h23m5.198574282s) [noise=true , address=193.149.164.143:4697 , piece_id=C2JF73NKW64HNXGS5BJTVZRCAIVWKZBGDGUZFJMDUWW3RBBXJDSA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF
Dec 29 12:52:05.327500265 (573h23m5.1985833s) [noise=true , address=193.149.164.143:4697 , piece_id=C2JF73NKW64HNXGS5BJTVZRCAIVWKZBGDGUZFJMDUWW3RBBXJDSA, seg_pos={0 0}, upload=502680]: Long tail cancellation due to optimal threshold
Dec 29 12:52:05.327548276 (573h23m5.198631321s) [seg_pos={0 0}, upload=502680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 12:52:05.327736143 (573h23m5.198819188s) [upload=502680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 12:52:05.327744139 (573h23m5.198827174s) [upload=502680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 12:52:05.624748327 (573h23m5.495831362s) [upload=502680]: Done uploading segments...
Dec 29 12:52:05.624772935 (573h23m5.49585597s) [upload=502680]: Upload finished. err: <nil>
Duration: 47.139962474s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.............................X......................X....X.X] duration:
30.001960083s trace:
https://jaeger.dev.storj.io/trace/141b6edfc3103b12
FAIL (2025-12-29 13:09:03 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001982716s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
52 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXX.X.XX.XXX.XX.XXXXXXX.X] duration:
5.001188414s trace:
https://jaeger.dev.storj.io/trace/2b04cfad43edb844
FAIL (2025-12-29 13:08:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001213473s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..............................X......................X....X.] duration:
17.306455163s trace:
https://jaeger.dev.storj.io/trace/7eb947425895541e
FAIL (2025-12-29 13:08:22 UTC)
orbiter warning: context deadline exceeded
Duration: 17.306478136s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.X..........X......................X.....X..................] duration:
43.12394169s trace:
https://jaeger.dev.storj.io/trace/213826fef0d4f1dd
FAIL (2025-12-29 13:08:10 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
c 29 12:52:02.421331209 (573h23m2.292414254s) [noise=true , address=145.79.227.48:4334 , piece_id=MFEYVH6L2DBOS7Z444QFOQDVQCQS5MG3M32335L7ZCN4MXVH5L2A, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.421480444 (573h23m2.292563479s) [noise=true , address=77.90.31.17:4799 , piece_id=DFRKPKCVCIPZZJB6QDU64E5JGJIIKXDEO5MJAA4X5V6N3ZTEGTFA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.422126223 (573h23m2.293209268s) [noise=true , address=77.90.31.14:4676 , piece_id=RGKFUGBFBMHXICXNPPCFXNFHOZ5EXIFPT54PBT2EA5AS2ZC5OTGA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.422527958 (573h23m2.293611013s) [noise=true , address=145.79.227.17:4980 , piece_id=D3CAP7TOKGQ2NVULNTN3WSB2OQKOLVWN7TFGLPOLLJPPMHHZR7NQ, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.422668796 (573h23m2.293751841s) [noise=true , address=193.149.164.143:4697 , piece_id=C2JF73NKW64HNXGS5BJTVZRCAIVWKZBGDGUZFJMDUWW3RBBXJDSA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.422966322 (573h23m2.294049357s) [noise=true , address=77.90.31.19:4863 , piece_id=XLK3SHF2GNV625H7RC5SGL67TKKBAJREGPTBCEJDX3QGD6H63GMA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.423193946 (573h23m2.294276981s) [noise=true , address=77.90.31.18:4829 , piece_id=RG37QNRSJFFED3XOPAEOHGJNJ45MEMUON3R3MB5ASEXS4A4EQ42Q, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.423513825 (573h23m2.29459687s) [noise=true , address=145.79.227.43:4166 , piece_id=QPEP6OT7WAFMRNNI6XVL25H3FAJ76FCQXVSEACOTV6XCYIZZSVPA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.423810040 (573h23m2.294893085s) [noise=true , address=77.90.31.10:4545 , piece_id=NE3YEJJJKKOQSM2JKO5PFTPJH7SG65UZGR3QJP5JZXOUF3FN32YQ, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.424441803 (573h23m2.295524868s) [noise=true , address=64.49.13.169:4708 , piece_id=5ORIIBIW2GPJRDD5C7IBQDPKPS5SJBTSVR6SGC6NKX36DTTGAYGA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.425051363 (573h23m2.296134408s) [noise=true , address=45.91.116.5:4323 , piece_id=7FAJKQP23A6U6SXPWVVAFVIHQA54JSPERWMCSONK5NYNHXRDLF3A, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.425681483 (573h23m2.296764528s) [noise=true , address=145.79.227.7:4303 , piece_id=MWIQ7KUZ534TFKQFMQADMBPKUFDGNFFK3JJEH3KF2CT6IQGHQRKQ, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.426480093 (573h23m2.297563138s) [noise=true , address=145.79.227.33:4877 , piece_id=IFC4ANBQU7ORCMSTYAAG3QKE4S2JPCWKDD47P5Z57V6TWAR6GFTA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.427077340 (573h23m2.298160385s) [noise=true , address=145.79.227.34:4937 , piece_id=2EYFRE3YDBIZK7YORZMWRETOEHJGDDKJVZBFW36FU6LX6PHRTDIA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.427529692 (573h23m2.298612737s) [noise=true , address=145.79.227.30:4793 , piece_id=AGUHSTJNNNRAAP5TKWIONYBZDKPJDJEQX7RX5GEGWN6TG4RELAMQ, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.428130125 (573h23m2.29921317s) [noise=true , address=77.90.31.16:4747 , piece_id=5TQY6C4SF6JQT6KTSEMOSY65TLPLUVCDTNVJ6U2QYFMTYP2G73VA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:03.783583605 (573h23m3.65466665s) [upload=502680]: Next returned nil segment
Dec 29 12:52:03.783597080 (573h23m3.654680125s) [upload=502680]: Waiting for error group managing segments...
Dec 29 12:52:04.132513046 (573h23m4.003596091s) [noise=true , address=64.49.13.137:4494 , piece_id=K4TIPJCGNFYIMSQKXQAWABJLIHHTAS3F36I723BHTJ7YG4HUKIIQ, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.271420967 (573h23m4.142504012s) [noise=true , address=145.79.227.7:4303 , piece_id=MWIQ7KUZ534TFKQFMQADMBPKUFDGNFFK3JJEH3KF2CT6IQGHQRKQ, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.275668766 (573h23m4.146751811s) [noise=true , address=145.79.227.48:4334 , piece_id=MFEYVH6L2DBOS7Z444QFOQDVQCQS5MG3M32335L7ZCN4MXVH5L2A, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.298256927 (573h23m4.169339972s) [noise=true , address=145.79.227.17:4980 , piece_id=D3CAP7TOKGQ2NVULNTN3WSB2OQKOLVWN7TFGLPOLLJPPMHHZR7NQ, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.487163872 (573h23m4.358246917s) [noise=true , address=77.90.31.19:4863 , piece_id=XLK3SHF2GNV625H7RC5SGL67TKKBAJREGPTBCEJDX3QGD6H63GMA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.563458298 (573h23m4.434541343s) [noise=true , address=45.91.116.5:4323 , piece_id=7FAJKQP23A6U6SXPWVVAFVIHQA54JSPERWMCSONK5NYNHXRDLF3A, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.572838760 (573h23m4.443921795s) [noise=true , address=77.90.31.10:4545 , piece_id=NE3YEJJJKKOQSM2JKO5PFTPJH7SG65UZGR3QJP5JZXOUF3FN32YQ, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.580866937 (573h23m4.451949972s) [noise=true , address=77.90.31.14:4676 , piece_id=RGKFUGBFBMHXICXNPPCFXNFHOZ5EXIFPT54PBT2EA5AS2ZC5OTGA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.595018524 (573h23m4.466101569s) [noise=true , address=77.90.31.17:4799 , piece_id=DFRKPKCVCIPZZJB6QDU64E5JGJIIKXDEO5MJAA4X5V6N3ZTEGTFA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:05.183831178 (573h23m5.054914213s) [noise=true , address=145.79.227.30:4793 , piece_id=AGUHSTJNNNRAAP5TKWIONYBZDKPJDJEQX7RX5GEGWN6TG4RELAMQ, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:05.242293548 (573h23m5.113376593s) [noise=true , address=145.79.227.33:4877 , piece_id=IFC4ANBQU7ORCMSTYAAG3QKE4S2JPCWKDD47P5Z57V6TWAR6GFTA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:05.255836156 (573h23m5.126919201s) [noise=true , address=145.79.227.34:4937 , piece_id=2EYFRE3YDBIZK7YORZMWRETOEHJGDDKJVZBFW36FU6LX6PHRTDIA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:05.305203848 (573h23m5.176286883s) [noise=true , address=145.79.227.40:4058 , piece_id=EN2T5WMAEV3PVNAOWLMN4BUIJSAXRMAIQBQ6PKQ7PWPATZLYYBIQ, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:05.326720429 (573h23m5.197803474s) [noise=true , address=145.79.227.43:4166 , piece_id=QPEP6OT7WAFMRNNI6XVL25H3FAJ76FCQXVSEACOTV6XCYIZZSVPA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:05.326747741 (573h23m5.197830786s) [seg_pos={0 0}, upload=502680]: Segment reached optimal threshold of 14 pieces.
Dec 29 12:52:05.327021833 (573h23m5.198104889s) [noise=true , address=77.90.31.18:4829 , piece_id=RG37QNRSJFFED3XOPAEOHGJNJ45MEMUON3R3MB5ASEXS4A4EQ42Q, seg_pos={0 0}, upload=502680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo): protocol: expected piece hash; context canceled; EOF
Dec 29 12:52:05.327031712 (573h23m5.198114757s) [noise=true , address=77.90.31.18:4829 , piece_id=RG37QNRSJFFED3XOPAEOHGJNJ45MEMUON3R3MB5ASEXS4A4EQ42Q, seg_pos={0 0}, upload=502680]: Long tail cancellation due to optimal threshold
Dec 29 12:52:05.327058794 (573h23m5.198141849s) [noise=true , address=64.49.13.169:4708 , piece_id=5ORIIBIW2GPJRDD5C7IBQDPKPS5SJBTSVR6SGC6NKX36DTTGAYGA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:40464->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:40464->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40464->64.49.13.169:4708: use of closed network connection
Dec 29 12:52:05.327066769 (573h23m5.198149814s) [noise=true , address=64.49.13.169:4708 , piece_id=5ORIIBIW2GPJRDD5C7IBQDPKPS5SJBTSVR6SGC6NKX36DTTGAYGA, seg_pos={0 0}, upload=502680]: Long tail cancellation due to optimal threshold
Dec 29 12:52:05.327203149 (573h23m5.198286204s) [noise=true , address=77.90.31.16:4747 , piece_id=5TQY6C4SF6JQT6KTSEMOSY65TLPLUVCDTNVJ6U2QYFMTYP2G73VA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2): protocol: expected piece hash; context canceled; EOF
Dec 29 12:52:05.327213137 (573h23m5.198296182s) [noise=true , address=77.90.31.16:4747 , piece_id=5TQY6C4SF6JQT6KTSEMOSY65TLPLUVCDTNVJ6U2QYFMTYP2G73VA, seg_pos={0 0}, upload=502680]: Long tail cancellation due to optimal threshold
Dec 29 12:52:05.327491227 (573h23m5.198574282s) [noise=true , address=193.149.164.143:4697 , piece_id=C2JF73NKW64HNXGS5BJTVZRCAIVWKZBGDGUZFJMDUWW3RBBXJDSA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF
Dec 29 12:52:05.327500265 (573h23m5.1985833s) [noise=true , address=193.149.164.143:4697 , piece_id=C2JF73NKW64HNXGS5BJTVZRCAIVWKZBGDGUZFJMDUWW3RBBXJDSA, seg_pos={0 0}, upload=502680]: Long tail cancellation due to optimal threshold
Dec 29 12:52:05.327548276 (573h23m5.198631321s) [seg_pos={0 0}, upload=502680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 12:52:05.327736143 (573h23m5.198819188s) [upload=502680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 12:52:05.327744139 (573h23m5.198827174s) [upload=502680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 12:52:05.624748327 (573h23m5.495831362s) [upload=502680]: Done uploading segments...
Dec 29 12:52:05.624772935 (573h23m5.49585597s) [upload=502680]: Upload finished. err: <nil>
Duration: 43.123959694s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
27 out of 60 is failed (threshold=3) [..X...X..X....XXXX..XX..X...XX.X.XX...X.X.X.XXX.X.X.X.XX..X.] duration:
5.684508784s trace:
https://jaeger.dev.storj.io/trace/4c200db05973ca5f
FAIL (2025-12-29 13:08:02 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 264PBWLVONJQKLA25YYLFXIRKVIFGZDORXARMH6JQJB55UQWNE3Q) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: 7RWE4O5PNR7CVPUTKIZJ2CBEGYETSSB3PKYJWSR2TPF4OSZEOE4A) context deadline exceeded
======LOGS======
Dec 29 13:06:52.609056454 (573h37m52.480139489s) [upload=502903]: Starting upload
Dec 29 13:06:52.609129503 (573h37m52.480212538s) [upload=502903]: Done starting upload
Dec 29 13:06:52.609364982 (573h37m52.480448027s) [upload=502903]: Uploading segments...
Dec 29 13:06:52.609415769 (573h37m52.480498804s) [upload=502903]: Got next segment. Inline: false
Dec 29 13:06:52.609448331 (573h37m52.480531366s) [upload=502903]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 13:06:52.609454082 (573h37m52.480537127s) [upload=502903]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 13:06:52.860745026 (573h37m52.731828061s) [seg_pos={0 0}, upload=502903]: Begin upload segment...
Dec 29 13:06:52.860805712 (573h37m52.731888757s) [seg_pos={0 0}, upload=502903]: Done begin upload segment.
Dec 29 13:06:52.860893789 (573h37m52.731976824s) [noise=true , address=145.79.227.48:4303 , piece_id=YMMCQ7KAUGHW654GA4AQXHG3UFGLLV63GWIEX7SETCNATJQTRJRA, seg_pos={0 0}, upload=502903]: Uploading piece...
Dec 29 13:06:52.860910361 (573h37m52.731993397s) [noise=true , address=145.79.227.42:4123 , piece_id=JPOKD2MZBMRT6CPHYPO6YLBZNV7DINRD4FIFVGZBGV2MPEKMWGDQ, seg_pos={0 0}, upload=502903]: Uploading piece...
Dec 29 13:06:52.862105447 (573h37m52.733188482s) [noise=true , address=145.79.227.11:4732 , piece_id=7TD6ACWMVINH24N5P3M5F4G63AN55I7SEPZYDAR2DAVLQW5NFFEA, seg_pos={0 0}, upload=502903]: Uploading piece...
Dec 29 13:06:52.862317340 (573h37m52.733400375s) [noise=true , address=45.91.116.11:4120 , piece_id=OSAKHMFFZOVGIIRSBNZGG2RIMTJFSOJL7J4WZMG6BR6DAQYZTATQ, seg_pos={0 0}, upload=502903]: Uploading piece...
Dec 29 13:06:52.863050767 (573h37m52.734133802s) [noise=true , address=45.91.116.13:4153 , piece_id=E2ZOKYGPRPIUFMGEOLESI5CKBU2Y5TKUQAA33VNFFC6LIOIPTXNQ, seg_pos={0 0}, upload=502903]: Uploading piece...
Dec 29 13:06:52.863303078 (573h37m52.734386123s) [noise=true , address=145.79.227.24:4582 , piece_id=GVRJTT2ZFQ5WQMLC5MKWRFLDZF6YIHOWCT5U2IEVRHCN3M25UEXA, seg_pos={0 0}, upload=502903]: Uploading piece...
Dec 29 13:06:52.863734830 (573h37m52.734817865s) [noise=true , address=45.91.116.8:4047 , piece_id=264PBWLVONJQKLA25YYLFXIRKVIFGZDORXARMH6JQJB55UQWNE3Q, seg_pos={0 0}, upload=502903]: Uploading piece...
Dec 29 13:06:52.864000646 (573h37m52.735083681s) [noise=true , address=145.79.227.5:4221 , piece_id=MWSBEASVJEERUMBS44UDEWPLFXP6VFM7CENE466JS7NXLPZAU55A, seg_pos={0 0}, upload=502903]: Uploading piece...
Dec 29 13:06:52.864547046 (573h37m52.735630091s) [noise=true , address=145.79.227.52:4446 , piece_id=RGI5FH336JDSOEQ4MA6QDUV7JDOVF3FF6ZBE636R63BBKGYLQVBA, seg_pos={0 0}, upload=502903]: Uploading piece...
Dec 29 13:06:52.864789768 (573h37m52.735872813s) [noise=true , address=145.79.227.14:4837 , piece_id=3SK2BR6QCYX5KDF7PSDMMAUT7DPXE7LXD7UOP7BRVYWM5XLXNS3A, seg_pos={0 0}, upload=502903]: Uploading piece...
Dec 29 13:06:52.865377638 (573h37m52.736460683s) [noise=true , address=145.79.227.32:4834 , piece_id=VPBEXJP3FHQZGV6GQXP4Z7G3GF3ANE7VL6DWCZMFWNGY3XTRKMOQ, seg_pos={0 0}, upload=502903]: Uploading piece...
Dec 29 13:06:52.865603438 (573h37m52.736686483s) [noise=true , address=145.79.227.47:4267 , piece_id=K5HV57THWJXE3BLDQZ56JLMMMJWH22Q5GDYBVGEILM5CIOLLAJDQ, seg_pos={0 0}, upload=502903]: Uploading piece...
Dec 29 13:06:52.866516577 (573h37m52.737599612s) [noise=true , address=45.91.116.9:4264 , piece_id=7RWE4O5PNR7CVPUTKIZJ2CBEGYETSSB3PKYJWSR2TPF4OSZEOE4A, seg_pos={0 0}, upload=502903]: Uploading piece...
Dec 29 13:06:52.866988555 (573h37m52.73807159s) [noise=true , address=145.79.227.3:4150 , piece_id=L3WIZXCSNHTXXCCGOAYHIJX4QR2VV5SNZPSXDAIN6C2XSJK7SYQQ, seg_pos={0 0}, upload=502903]: Uploading piece...
Dec 29 13:06:52.867485051 (573h37m52.738568096s) [noise=true , address=145.79.227.6:4259 , piece_id=YEJQLMOOFYVCF7MYR6S3M76DAWB2S5MCM4PIPUFCZ4GFOFDWPQNQ, seg_pos={0 0}, upload=502903]: Uploading piece...
Dec 29 13:06:52.870608217 (573h37m52.741691262s) [noise=true , address=145.79.227.50:4374 , piece_id=46XIG5WDCMGAKXM2OBHHM6C4QRZYCYF5PBK7HE2GSZCQUSOLGYGQ, seg_pos={0 0}, upload=502903]: Uploading piece...
Dec 29 13:06:54.558459243 (573h37m54.429542278s) [upload=502903]: Next returned nil segment
Dec 29 13:06:54.558477547 (573h37m54.429560582s) [upload=502903]: Waiting for error group managing segments...
Dec 29 13:06:54.839061270 (573h37m54.710144306s) [noise=true , address=145.79.227.24:4582 , piece_id=GVRJTT2ZFQ5WQMLC5MKWRFLDZF6YIHOWCT5U2IEVRHCN3M25UEXA, seg_pos={0 0}, upload=502903]: Done uploading piece. err: <nil>
Dec 29 13:06:54.850771358 (573h37m54.721854393s) [noise=true , address=145.79.227.6:4259 , piece_id=YEJQLMOOFYVCF7MYR6S3M76DAWB2S5MCM4PIPUFCZ4GFOFDWPQNQ, seg_pos={0 0}, upload=502903]: Done uploading piece. err: <nil>
Dec 29 13:06:54.852912395 (573h37m54.72399543s) [noise=true , address=145.79.227.32:4834 , piece_id=VPBEXJP3FHQZGV6GQXP4Z7G3GF3ANE7VL6DWCZMFWNGY3XTRKMOQ, seg_pos={0 0}, upload=502903]: Done uploading piece. err: <nil>
Dec 29 13:06:54.853863105 (573h37m54.72494615s) [noise=true , address=145.79.227.52:4446 , piece_id=RGI5FH336JDSOEQ4MA6QDUV7JDOVF3FF6ZBE636R63BBKGYLQVBA, seg_pos={0 0}, upload=502903]: Done uploading piece. err: <nil>
Dec 29 13:06:54.855852554 (573h37m54.726935589s) [noise=true , address=145.79.227.42:4123 , piece_id=JPOKD2MZBMRT6CPHYPO6YLBZNV7DINRD4FIFVGZBGV2MPEKMWGDQ, seg_pos={0 0}, upload=502903]: Done uploading piece. err: <nil>
Dec 29 13:06:54.857288949 (573h37m54.728371984s) [noise=true , address=145.79.227.14:4837 , piece_id=3SK2BR6QCYX5KDF7PSDMMAUT7DPXE7LXD7UOP7BRVYWM5XLXNS3A, seg_pos={0 0}, upload=502903]: Done uploading piece. err: <nil>
Dec 29 13:06:54.860509490 (573h37m54.731592526s) [noise=true , address=145.79.227.50:4374 , piece_id=46XIG5WDCMGAKXM2OBHHM6C4QRZYCYF5PBK7HE2GSZCQUSOLGYGQ, seg_pos={0 0}, upload=502903]: Done uploading piece. err: <nil>
Dec 29 13:06:54.866805459 (573h37m54.737888504s) [noise=true , address=145.79.227.47:4267 , piece_id=K5HV57THWJXE3BLDQZ56JLMMMJWH22Q5GDYBVGEILM5CIOLLAJDQ, seg_pos={0 0}, upload=502903]: Done uploading piece. err: <nil>
Dec 29 13:06:54.869465223 (573h37m54.740548259s) [noise=true , address=145.79.227.48:4303 , piece_id=YMMCQ7KAUGHW654GA4AQXHG3UFGLLV63GWIEX7SETCNATJQTRJRA, seg_pos={0 0}, upload=502903]: Done uploading piece. err: <nil>
Dec 29 13:06:54.870629040 (573h37m54.741712085s) [noise=true , address=145.79.227.5:4221 , piece_id=MWSBEASVJEERUMBS44UDEWPLFXP6VFM7CENE466JS7NXLPZAU55A, seg_pos={0 0}, upload=502903]: Done uploading piece. err: <nil>
Dec 29 13:06:54.872724550 (573h37m54.743807595s) [noise=true , address=145.79.227.3:4150 , piece_id=L3WIZXCSNHTXXCCGOAYHIJX4QR2VV5SNZPSXDAIN6C2XSJK7SYQQ, seg_pos={0 0}, upload=502903]: Done uploading piece. err: <nil>
Dec 29 13:06:54.895051904 (573h37m54.766134949s) [noise=true , address=145.79.227.11:4732 , piece_id=7TD6ACWMVINH24N5P3M5F4G63AN55I7SEPZYDAR2DAVLQW5NFFEA, seg_pos={0 0}, upload=502903]: Done uploading piece. err: <nil>
Dec 29 13:06:55.224975106 (573h37m55.096058141s) [noise=true , address=45.91.116.8:4047 , piece_id=264PBWLVONJQKLA25YYLFXIRKVIFGZDORXARMH6JQJB55UQWNE3Q, seg_pos={0 0}, upload=502903]: Done uploading piece. err: <nil>
Dec 29 13:06:55.225000905 (573h37m55.09608395s) [seg_pos={0 0}, upload=502903]: Segment reached optimal threshold of 13 pieces.
Dec 29 13:06:55.225196398 (573h37m55.096279443s) [noise=true , address=45.91.116.9:4264 , piece_id=7RWE4O5PNR7CVPUTKIZJ2CBEGYETSSB3PKYJWSR2TPF4OSZEOE4A, seg_pos={0 0}, upload=502903]: Done uploading piece. err: <nil>
Dec 29 13:06:55.225385848 (573h37m55.096468923s) [noise=true , address=45.91.116.13:4153 , piece_id=E2ZOKYGPRPIUFMGEOLESI5CKBU2Y5TKUQAA33VNFFC6LIOIPTXNQ, seg_pos={0 0}, upload=502903]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:50514->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:50514->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50514->45.91.116.13:4153: use of closed network connection
Dec 29 13:06:55.225400807 (573h37m55.096483862s) [noise=true , address=45.91.116.13:4153 , piece_id=E2ZOKYGPRPIUFMGEOLESI5CKBU2Y5TKUQAA33VNFFC6LIOIPTXNQ, seg_pos={0 0}, upload=502903]: Long tail cancellation due to optimal threshold
Dec 29 13:06:55.225752978 (573h37m55.096836013s) [noise=true , address=45.91.116.11:4120 , piece_id=OSAKHMFFZOVGIIRSBNZGG2RIMTJFSOJL7J4WZMG6BR6DAQYZTATQ, seg_pos={0 0}, upload=502903]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF
Dec 29 13:06:55.225763357 (573h37m55.096846392s) [noise=true , address=45.91.116.11:4120 , piece_id=OSAKHMFFZOVGIIRSBNZGG2RIMTJFSOJL7J4WZMG6BR6DAQYZTATQ, seg_pos={0 0}, upload=502903]: Long tail cancellation due to optimal threshold
Dec 29 13:06:55.225928031 (573h37m55.097011076s) [seg_pos={0 0}, upload=502903]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil>
Dec 29 13:06:55.226155754 (573h37m55.097238819s) [upload=502903]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:06:55.226166846 (573h37m55.097249881s) [upload=502903]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:06:55.519504623 (573h37m55.390587668s) [upload=502903]: Done uploading segments...
Dec 29 13:06:55.519527116 (573h37m55.390610161s) [upload=502903]: Upload finished. err: <nil>
Duration: 5.684535084s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
52 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXX.X.XX.XXX.XX.XXXXXXX.] duration:
4.622131562s trace:
https://jaeger.dev.storj.io/trace/1d2f5669acbd6844
FAIL (2025-12-29 13:07:47 UTC)
orbiter warning: context deadline exceeded
Duration: 4.622158443s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...............................X......................X....X] duration:
30.000929426s trace:
https://jaeger.dev.storj.io/trace/7eb947425895541e
FAIL (2025-12-29 13:07:29 UTC)
orbiter warning: context deadline exceeded
Duration: 30.000948952s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
53 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXX.X.XX.XXX.XX.XXXXXXX] duration:
5.000596075s trace:
https://jaeger.dev.storj.io/trace/1d2f5669acbd6844
FAIL (2025-12-29 13:07:28 UTC)
orbiter warning: context deadline exceeded
Duration: 5.00062444s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [..X..........X......................X.....X.................] duration:
45.374198434s trace:
https://jaeger.dev.storj.io/trace/213826fef0d4f1dd
FAIL (2025-12-29 13:07:26 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
c 29 12:52:02.421331209 (573h23m2.292414254s) [noise=true , address=145.79.227.48:4334 , piece_id=MFEYVH6L2DBOS7Z444QFOQDVQCQS5MG3M32335L7ZCN4MXVH5L2A, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.421480444 (573h23m2.292563479s) [noise=true , address=77.90.31.17:4799 , piece_id=DFRKPKCVCIPZZJB6QDU64E5JGJIIKXDEO5MJAA4X5V6N3ZTEGTFA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.422126223 (573h23m2.293209268s) [noise=true , address=77.90.31.14:4676 , piece_id=RGKFUGBFBMHXICXNPPCFXNFHOZ5EXIFPT54PBT2EA5AS2ZC5OTGA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.422527958 (573h23m2.293611013s) [noise=true , address=145.79.227.17:4980 , piece_id=D3CAP7TOKGQ2NVULNTN3WSB2OQKOLVWN7TFGLPOLLJPPMHHZR7NQ, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.422668796 (573h23m2.293751841s) [noise=true , address=193.149.164.143:4697 , piece_id=C2JF73NKW64HNXGS5BJTVZRCAIVWKZBGDGUZFJMDUWW3RBBXJDSA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.422966322 (573h23m2.294049357s) [noise=true , address=77.90.31.19:4863 , piece_id=XLK3SHF2GNV625H7RC5SGL67TKKBAJREGPTBCEJDX3QGD6H63GMA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.423193946 (573h23m2.294276981s) [noise=true , address=77.90.31.18:4829 , piece_id=RG37QNRSJFFED3XOPAEOHGJNJ45MEMUON3R3MB5ASEXS4A4EQ42Q, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.423513825 (573h23m2.29459687s) [noise=true , address=145.79.227.43:4166 , piece_id=QPEP6OT7WAFMRNNI6XVL25H3FAJ76FCQXVSEACOTV6XCYIZZSVPA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.423810040 (573h23m2.294893085s) [noise=true , address=77.90.31.10:4545 , piece_id=NE3YEJJJKKOQSM2JKO5PFTPJH7SG65UZGR3QJP5JZXOUF3FN32YQ, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.424441803 (573h23m2.295524868s) [noise=true , address=64.49.13.169:4708 , piece_id=5ORIIBIW2GPJRDD5C7IBQDPKPS5SJBTSVR6SGC6NKX36DTTGAYGA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.425051363 (573h23m2.296134408s) [noise=true , address=45.91.116.5:4323 , piece_id=7FAJKQP23A6U6SXPWVVAFVIHQA54JSPERWMCSONK5NYNHXRDLF3A, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.425681483 (573h23m2.296764528s) [noise=true , address=145.79.227.7:4303 , piece_id=MWIQ7KUZ534TFKQFMQADMBPKUFDGNFFK3JJEH3KF2CT6IQGHQRKQ, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.426480093 (573h23m2.297563138s) [noise=true , address=145.79.227.33:4877 , piece_id=IFC4ANBQU7ORCMSTYAAG3QKE4S2JPCWKDD47P5Z57V6TWAR6GFTA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.427077340 (573h23m2.298160385s) [noise=true , address=145.79.227.34:4937 , piece_id=2EYFRE3YDBIZK7YORZMWRETOEHJGDDKJVZBFW36FU6LX6PHRTDIA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.427529692 (573h23m2.298612737s) [noise=true , address=145.79.227.30:4793 , piece_id=AGUHSTJNNNRAAP5TKWIONYBZDKPJDJEQX7RX5GEGWN6TG4RELAMQ, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.428130125 (573h23m2.29921317s) [noise=true , address=77.90.31.16:4747 , piece_id=5TQY6C4SF6JQT6KTSEMOSY65TLPLUVCDTNVJ6U2QYFMTYP2G73VA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:03.783583605 (573h23m3.65466665s) [upload=502680]: Next returned nil segment
Dec 29 12:52:03.783597080 (573h23m3.654680125s) [upload=502680]: Waiting for error group managing segments...
Dec 29 12:52:04.132513046 (573h23m4.003596091s) [noise=true , address=64.49.13.137:4494 , piece_id=K4TIPJCGNFYIMSQKXQAWABJLIHHTAS3F36I723BHTJ7YG4HUKIIQ, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.271420967 (573h23m4.142504012s) [noise=true , address=145.79.227.7:4303 , piece_id=MWIQ7KUZ534TFKQFMQADMBPKUFDGNFFK3JJEH3KF2CT6IQGHQRKQ, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.275668766 (573h23m4.146751811s) [noise=true , address=145.79.227.48:4334 , piece_id=MFEYVH6L2DBOS7Z444QFOQDVQCQS5MG3M32335L7ZCN4MXVH5L2A, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.298256927 (573h23m4.169339972s) [noise=true , address=145.79.227.17:4980 , piece_id=D3CAP7TOKGQ2NVULNTN3WSB2OQKOLVWN7TFGLPOLLJPPMHHZR7NQ, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.487163872 (573h23m4.358246917s) [noise=true , address=77.90.31.19:4863 , piece_id=XLK3SHF2GNV625H7RC5SGL67TKKBAJREGPTBCEJDX3QGD6H63GMA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.563458298 (573h23m4.434541343s) [noise=true , address=45.91.116.5:4323 , piece_id=7FAJKQP23A6U6SXPWVVAFVIHQA54JSPERWMCSONK5NYNHXRDLF3A, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.572838760 (573h23m4.443921795s) [noise=true , address=77.90.31.10:4545 , piece_id=NE3YEJJJKKOQSM2JKO5PFTPJH7SG65UZGR3QJP5JZXOUF3FN32YQ, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.580866937 (573h23m4.451949972s) [noise=true , address=77.90.31.14:4676 , piece_id=RGKFUGBFBMHXICXNPPCFXNFHOZ5EXIFPT54PBT2EA5AS2ZC5OTGA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.595018524 (573h23m4.466101569s) [noise=true , address=77.90.31.17:4799 , piece_id=DFRKPKCVCIPZZJB6QDU64E5JGJIIKXDEO5MJAA4X5V6N3ZTEGTFA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:05.183831178 (573h23m5.054914213s) [noise=true , address=145.79.227.30:4793 , piece_id=AGUHSTJNNNRAAP5TKWIONYBZDKPJDJEQX7RX5GEGWN6TG4RELAMQ, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:05.242293548 (573h23m5.113376593s) [noise=true , address=145.79.227.33:4877 , piece_id=IFC4ANBQU7ORCMSTYAAG3QKE4S2JPCWKDD47P5Z57V6TWAR6GFTA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:05.255836156 (573h23m5.126919201s) [noise=true , address=145.79.227.34:4937 , piece_id=2EYFRE3YDBIZK7YORZMWRETOEHJGDDKJVZBFW36FU6LX6PHRTDIA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:05.305203848 (573h23m5.176286883s) [noise=true , address=145.79.227.40:4058 , piece_id=EN2T5WMAEV3PVNAOWLMN4BUIJSAXRMAIQBQ6PKQ7PWPATZLYYBIQ, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:05.326720429 (573h23m5.197803474s) [noise=true , address=145.79.227.43:4166 , piece_id=QPEP6OT7WAFMRNNI6XVL25H3FAJ76FCQXVSEACOTV6XCYIZZSVPA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:05.326747741 (573h23m5.197830786s) [seg_pos={0 0}, upload=502680]: Segment reached optimal threshold of 14 pieces.
Dec 29 12:52:05.327021833 (573h23m5.198104889s) [noise=true , address=77.90.31.18:4829 , piece_id=RG37QNRSJFFED3XOPAEOHGJNJ45MEMUON3R3MB5ASEXS4A4EQ42Q, seg_pos={0 0}, upload=502680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo): protocol: expected piece hash; context canceled; EOF
Dec 29 12:52:05.327031712 (573h23m5.198114757s) [noise=true , address=77.90.31.18:4829 , piece_id=RG37QNRSJFFED3XOPAEOHGJNJ45MEMUON3R3MB5ASEXS4A4EQ42Q, seg_pos={0 0}, upload=502680]: Long tail cancellation due to optimal threshold
Dec 29 12:52:05.327058794 (573h23m5.198141849s) [noise=true , address=64.49.13.169:4708 , piece_id=5ORIIBIW2GPJRDD5C7IBQDPKPS5SJBTSVR6SGC6NKX36DTTGAYGA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:40464->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:40464->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40464->64.49.13.169:4708: use of closed network connection
Dec 29 12:52:05.327066769 (573h23m5.198149814s) [noise=true , address=64.49.13.169:4708 , piece_id=5ORIIBIW2GPJRDD5C7IBQDPKPS5SJBTSVR6SGC6NKX36DTTGAYGA, seg_pos={0 0}, upload=502680]: Long tail cancellation due to optimal threshold
Dec 29 12:52:05.327203149 (573h23m5.198286204s) [noise=true , address=77.90.31.16:4747 , piece_id=5TQY6C4SF6JQT6KTSEMOSY65TLPLUVCDTNVJ6U2QYFMTYP2G73VA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2): protocol: expected piece hash; context canceled; EOF
Dec 29 12:52:05.327213137 (573h23m5.198296182s) [noise=true , address=77.90.31.16:4747 , piece_id=5TQY6C4SF6JQT6KTSEMOSY65TLPLUVCDTNVJ6U2QYFMTYP2G73VA, seg_pos={0 0}, upload=502680]: Long tail cancellation due to optimal threshold
Dec 29 12:52:05.327491227 (573h23m5.198574282s) [noise=true , address=193.149.164.143:4697 , piece_id=C2JF73NKW64HNXGS5BJTVZRCAIVWKZBGDGUZFJMDUWW3RBBXJDSA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF
Dec 29 12:52:05.327500265 (573h23m5.1985833s) [noise=true , address=193.149.164.143:4697 , piece_id=C2JF73NKW64HNXGS5BJTVZRCAIVWKZBGDGUZFJMDUWW3RBBXJDSA, seg_pos={0 0}, upload=502680]: Long tail cancellation due to optimal threshold
Dec 29 12:52:05.327548276 (573h23m5.198631321s) [seg_pos={0 0}, upload=502680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 12:52:05.327736143 (573h23m5.198819188s) [upload=502680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 12:52:05.327744139 (573h23m5.198827174s) [upload=502680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 12:52:05.624748327 (573h23m5.495831362s) [upload=502680]: Done uploading segments...
Dec 29 12:52:05.624772935 (573h23m5.49585597s) [upload=502680]: Upload finished. err: <nil>
Duration: 45.374235153s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
27 out of 60 is failed (threshold=3) [...X...X..X....XXXX..XX..X...XX.X.XX...X.X.X.XXX.X.X.X.XX..X] duration:
30.003001542s trace:
https://jaeger.dev.storj.io/trace/4c200db05973ca5f
FAIL (2025-12-29 13:07:21 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: 264PBWLVONJQKLA25YYLFXIRKVIFGZDORXARMH6JQJB55UQWNE3Q) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: 7RWE4O5PNR7CVPUTKIZJ2CBEGYETSSB3PKYJWSR2TPF4OSZEOE4A) context deadline exceeded
======LOGS======
Dec 29 13:06:52.609056454 (573h37m52.480139489s) [upload=502903]: Starting upload
Dec 29 13:06:52.609129503 (573h37m52.480212538s) [upload=502903]: Done starting upload
Dec 29 13:06:52.609364982 (573h37m52.480448027s) [upload=502903]: Uploading segments...
Dec 29 13:06:52.609415769 (573h37m52.480498804s) [upload=502903]: Got next segment. Inline: false
Dec 29 13:06:52.609448331 (573h37m52.480531366s) [upload=502903]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 13:06:52.609454082 (573h37m52.480537127s) [upload=502903]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 13:06:52.860745026 (573h37m52.731828061s) [seg_pos={0 0}, upload=502903]: Begin upload segment...
Dec 29 13:06:52.860805712 (573h37m52.731888757s) [seg_pos={0 0}, upload=502903]: Done begin upload segment.
Dec 29 13:06:52.860893789 (573h37m52.731976824s) [noise=true , address=145.79.227.48:4303 , piece_id=YMMCQ7KAUGHW654GA4AQXHG3UFGLLV63GWIEX7SETCNATJQTRJRA, seg_pos={0 0}, upload=502903]: Uploading piece...
Dec 29 13:06:52.860910361 (573h37m52.731993397s) [noise=true , address=145.79.227.42:4123 , piece_id=JPOKD2MZBMRT6CPHYPO6YLBZNV7DINRD4FIFVGZBGV2MPEKMWGDQ, seg_pos={0 0}, upload=502903]: Uploading piece...
Dec 29 13:06:52.862105447 (573h37m52.733188482s) [noise=true , address=145.79.227.11:4732 , piece_id=7TD6ACWMVINH24N5P3M5F4G63AN55I7SEPZYDAR2DAVLQW5NFFEA, seg_pos={0 0}, upload=502903]: Uploading piece...
Dec 29 13:06:52.862317340 (573h37m52.733400375s) [noise=true , address=45.91.116.11:4120 , piece_id=OSAKHMFFZOVGIIRSBNZGG2RIMTJFSOJL7J4WZMG6BR6DAQYZTATQ, seg_pos={0 0}, upload=502903]: Uploading piece...
Dec 29 13:06:52.863050767 (573h37m52.734133802s) [noise=true , address=45.91.116.13:4153 , piece_id=E2ZOKYGPRPIUFMGEOLESI5CKBU2Y5TKUQAA33VNFFC6LIOIPTXNQ, seg_pos={0 0}, upload=502903]: Uploading piece...
Dec 29 13:06:52.863303078 (573h37m52.734386123s) [noise=true , address=145.79.227.24:4582 , piece_id=GVRJTT2ZFQ5WQMLC5MKWRFLDZF6YIHOWCT5U2IEVRHCN3M25UEXA, seg_pos={0 0}, upload=502903]: Uploading piece...
Dec 29 13:06:52.863734830 (573h37m52.734817865s) [noise=true , address=45.91.116.8:4047 , piece_id=264PBWLVONJQKLA25YYLFXIRKVIFGZDORXARMH6JQJB55UQWNE3Q, seg_pos={0 0}, upload=502903]: Uploading piece...
Dec 29 13:06:52.864000646 (573h37m52.735083681s) [noise=true , address=145.79.227.5:4221 , piece_id=MWSBEASVJEERUMBS44UDEWPLFXP6VFM7CENE466JS7NXLPZAU55A, seg_pos={0 0}, upload=502903]: Uploading piece...
Dec 29 13:06:52.864547046 (573h37m52.735630091s) [noise=true , address=145.79.227.52:4446 , piece_id=RGI5FH336JDSOEQ4MA6QDUV7JDOVF3FF6ZBE636R63BBKGYLQVBA, seg_pos={0 0}, upload=502903]: Uploading piece...
Dec 29 13:06:52.864789768 (573h37m52.735872813s) [noise=true , address=145.79.227.14:4837 , piece_id=3SK2BR6QCYX5KDF7PSDMMAUT7DPXE7LXD7UOP7BRVYWM5XLXNS3A, seg_pos={0 0}, upload=502903]: Uploading piece...
Dec 29 13:06:52.865377638 (573h37m52.736460683s) [noise=true , address=145.79.227.32:4834 , piece_id=VPBEXJP3FHQZGV6GQXP4Z7G3GF3ANE7VL6DWCZMFWNGY3XTRKMOQ, seg_pos={0 0}, upload=502903]: Uploading piece...
Dec 29 13:06:52.865603438 (573h37m52.736686483s) [noise=true , address=145.79.227.47:4267 , piece_id=K5HV57THWJXE3BLDQZ56JLMMMJWH22Q5GDYBVGEILM5CIOLLAJDQ, seg_pos={0 0}, upload=502903]: Uploading piece...
Dec 29 13:06:52.866516577 (573h37m52.737599612s) [noise=true , address=45.91.116.9:4264 , piece_id=7RWE4O5PNR7CVPUTKIZJ2CBEGYETSSB3PKYJWSR2TPF4OSZEOE4A, seg_pos={0 0}, upload=502903]: Uploading piece...
Dec 29 13:06:52.866988555 (573h37m52.73807159s) [noise=true , address=145.79.227.3:4150 , piece_id=L3WIZXCSNHTXXCCGOAYHIJX4QR2VV5SNZPSXDAIN6C2XSJK7SYQQ, seg_pos={0 0}, upload=502903]: Uploading piece...
Dec 29 13:06:52.867485051 (573h37m52.738568096s) [noise=true , address=145.79.227.6:4259 , piece_id=YEJQLMOOFYVCF7MYR6S3M76DAWB2S5MCM4PIPUFCZ4GFOFDWPQNQ, seg_pos={0 0}, upload=502903]: Uploading piece...
Dec 29 13:06:52.870608217 (573h37m52.741691262s) [noise=true , address=145.79.227.50:4374 , piece_id=46XIG5WDCMGAKXM2OBHHM6C4QRZYCYF5PBK7HE2GSZCQUSOLGYGQ, seg_pos={0 0}, upload=502903]: Uploading piece...
Dec 29 13:06:54.558459243 (573h37m54.429542278s) [upload=502903]: Next returned nil segment
Dec 29 13:06:54.558477547 (573h37m54.429560582s) [upload=502903]: Waiting for error group managing segments...
Dec 29 13:06:54.839061270 (573h37m54.710144306s) [noise=true , address=145.79.227.24:4582 , piece_id=GVRJTT2ZFQ5WQMLC5MKWRFLDZF6YIHOWCT5U2IEVRHCN3M25UEXA, seg_pos={0 0}, upload=502903]: Done uploading piece. err: <nil>
Dec 29 13:06:54.850771358 (573h37m54.721854393s) [noise=true , address=145.79.227.6:4259 , piece_id=YEJQLMOOFYVCF7MYR6S3M76DAWB2S5MCM4PIPUFCZ4GFOFDWPQNQ, seg_pos={0 0}, upload=502903]: Done uploading piece. err: <nil>
Dec 29 13:06:54.852912395 (573h37m54.72399543s) [noise=true , address=145.79.227.32:4834 , piece_id=VPBEXJP3FHQZGV6GQXP4Z7G3GF3ANE7VL6DWCZMFWNGY3XTRKMOQ, seg_pos={0 0}, upload=502903]: Done uploading piece. err: <nil>
Dec 29 13:06:54.853863105 (573h37m54.72494615s) [noise=true , address=145.79.227.52:4446 , piece_id=RGI5FH336JDSOEQ4MA6QDUV7JDOVF3FF6ZBE636R63BBKGYLQVBA, seg_pos={0 0}, upload=502903]: Done uploading piece. err: <nil>
Dec 29 13:06:54.855852554 (573h37m54.726935589s) [noise=true , address=145.79.227.42:4123 , piece_id=JPOKD2MZBMRT6CPHYPO6YLBZNV7DINRD4FIFVGZBGV2MPEKMWGDQ, seg_pos={0 0}, upload=502903]: Done uploading piece. err: <nil>
Dec 29 13:06:54.857288949 (573h37m54.728371984s) [noise=true , address=145.79.227.14:4837 , piece_id=3SK2BR6QCYX5KDF7PSDMMAUT7DPXE7LXD7UOP7BRVYWM5XLXNS3A, seg_pos={0 0}, upload=502903]: Done uploading piece. err: <nil>
Dec 29 13:06:54.860509490 (573h37m54.731592526s) [noise=true , address=145.79.227.50:4374 , piece_id=46XIG5WDCMGAKXM2OBHHM6C4QRZYCYF5PBK7HE2GSZCQUSOLGYGQ, seg_pos={0 0}, upload=502903]: Done uploading piece. err: <nil>
Dec 29 13:06:54.866805459 (573h37m54.737888504s) [noise=true , address=145.79.227.47:4267 , piece_id=K5HV57THWJXE3BLDQZ56JLMMMJWH22Q5GDYBVGEILM5CIOLLAJDQ, seg_pos={0 0}, upload=502903]: Done uploading piece. err: <nil>
Dec 29 13:06:54.869465223 (573h37m54.740548259s) [noise=true , address=145.79.227.48:4303 , piece_id=YMMCQ7KAUGHW654GA4AQXHG3UFGLLV63GWIEX7SETCNATJQTRJRA, seg_pos={0 0}, upload=502903]: Done uploading piece. err: <nil>
Dec 29 13:06:54.870629040 (573h37m54.741712085s) [noise=true , address=145.79.227.5:4221 , piece_id=MWSBEASVJEERUMBS44UDEWPLFXP6VFM7CENE466JS7NXLPZAU55A, seg_pos={0 0}, upload=502903]: Done uploading piece. err: <nil>
Dec 29 13:06:54.872724550 (573h37m54.743807595s) [noise=true , address=145.79.227.3:4150 , piece_id=L3WIZXCSNHTXXCCGOAYHIJX4QR2VV5SNZPSXDAIN6C2XSJK7SYQQ, seg_pos={0 0}, upload=502903]: Done uploading piece. err: <nil>
Dec 29 13:06:54.895051904 (573h37m54.766134949s) [noise=true , address=145.79.227.11:4732 , piece_id=7TD6ACWMVINH24N5P3M5F4G63AN55I7SEPZYDAR2DAVLQW5NFFEA, seg_pos={0 0}, upload=502903]: Done uploading piece. err: <nil>
Dec 29 13:06:55.224975106 (573h37m55.096058141s) [noise=true , address=45.91.116.8:4047 , piece_id=264PBWLVONJQKLA25YYLFXIRKVIFGZDORXARMH6JQJB55UQWNE3Q, seg_pos={0 0}, upload=502903]: Done uploading piece. err: <nil>
Dec 29 13:06:55.225000905 (573h37m55.09608395s) [seg_pos={0 0}, upload=502903]: Segment reached optimal threshold of 13 pieces.
Dec 29 13:06:55.225196398 (573h37m55.096279443s) [noise=true , address=45.91.116.9:4264 , piece_id=7RWE4O5PNR7CVPUTKIZJ2CBEGYETSSB3PKYJWSR2TPF4OSZEOE4A, seg_pos={0 0}, upload=502903]: Done uploading piece. err: <nil>
Dec 29 13:06:55.225385848 (573h37m55.096468923s) [noise=true , address=45.91.116.13:4153 , piece_id=E2ZOKYGPRPIUFMGEOLESI5CKBU2Y5TKUQAA33VNFFC6LIOIPTXNQ, seg_pos={0 0}, upload=502903]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:50514->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:50514->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50514->45.91.116.13:4153: use of closed network connection
Dec 29 13:06:55.225400807 (573h37m55.096483862s) [noise=true , address=45.91.116.13:4153 , piece_id=E2ZOKYGPRPIUFMGEOLESI5CKBU2Y5TKUQAA33VNFFC6LIOIPTXNQ, seg_pos={0 0}, upload=502903]: Long tail cancellation due to optimal threshold
Dec 29 13:06:55.225752978 (573h37m55.096836013s) [noise=true , address=45.91.116.11:4120 , piece_id=OSAKHMFFZOVGIIRSBNZGG2RIMTJFSOJL7J4WZMG6BR6DAQYZTATQ, seg_pos={0 0}, upload=502903]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF
Dec 29 13:06:55.225763357 (573h37m55.096846392s) [noise=true , address=45.91.116.11:4120 , piece_id=OSAKHMFFZOVGIIRSBNZGG2RIMTJFSOJL7J4WZMG6BR6DAQYZTATQ, seg_pos={0 0}, upload=502903]: Long tail cancellation due to optimal threshold
Dec 29 13:06:55.225928031 (573h37m55.097011076s) [seg_pos={0 0}, upload=502903]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil>
Dec 29 13:06:55.226155754 (573h37m55.097238819s) [upload=502903]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 13:06:55.226166846 (573h37m55.097249881s) [upload=502903]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 13:06:55.519504623 (573h37m55.390587668s) [upload=502903]: Done uploading segments...
Dec 29 13:06:55.519527116 (573h37m55.390610161s) [upload=502903]: Upload finished. err: <nil>
Duration: 30.003024405s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
53 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX.XXXXXXX.X.XX.XXX.XX.XXXXXX] duration:
5.001787571s trace:
https://jaeger.dev.storj.io/trace/78d7d52f425c6448
FAIL (2025-12-29 13:07:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001815254s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [...X..........X......................X.....X................] duration:
50.065628498s trace:
https://jaeger.dev.storj.io/trace/213826fef0d4f1dd
FAIL (2025-12-29 13:06:39 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
c 29 12:52:02.421331209 (573h23m2.292414254s) [noise=true , address=145.79.227.48:4334 , piece_id=MFEYVH6L2DBOS7Z444QFOQDVQCQS5MG3M32335L7ZCN4MXVH5L2A, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.421480444 (573h23m2.292563479s) [noise=true , address=77.90.31.17:4799 , piece_id=DFRKPKCVCIPZZJB6QDU64E5JGJIIKXDEO5MJAA4X5V6N3ZTEGTFA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.422126223 (573h23m2.293209268s) [noise=true , address=77.90.31.14:4676 , piece_id=RGKFUGBFBMHXICXNPPCFXNFHOZ5EXIFPT54PBT2EA5AS2ZC5OTGA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.422527958 (573h23m2.293611013s) [noise=true , address=145.79.227.17:4980 , piece_id=D3CAP7TOKGQ2NVULNTN3WSB2OQKOLVWN7TFGLPOLLJPPMHHZR7NQ, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.422668796 (573h23m2.293751841s) [noise=true , address=193.149.164.143:4697 , piece_id=C2JF73NKW64HNXGS5BJTVZRCAIVWKZBGDGUZFJMDUWW3RBBXJDSA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.422966322 (573h23m2.294049357s) [noise=true , address=77.90.31.19:4863 , piece_id=XLK3SHF2GNV625H7RC5SGL67TKKBAJREGPTBCEJDX3QGD6H63GMA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.423193946 (573h23m2.294276981s) [noise=true , address=77.90.31.18:4829 , piece_id=RG37QNRSJFFED3XOPAEOHGJNJ45MEMUON3R3MB5ASEXS4A4EQ42Q, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.423513825 (573h23m2.29459687s) [noise=true , address=145.79.227.43:4166 , piece_id=QPEP6OT7WAFMRNNI6XVL25H3FAJ76FCQXVSEACOTV6XCYIZZSVPA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.423810040 (573h23m2.294893085s) [noise=true , address=77.90.31.10:4545 , piece_id=NE3YEJJJKKOQSM2JKO5PFTPJH7SG65UZGR3QJP5JZXOUF3FN32YQ, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.424441803 (573h23m2.295524868s) [noise=true , address=64.49.13.169:4708 , piece_id=5ORIIBIW2GPJRDD5C7IBQDPKPS5SJBTSVR6SGC6NKX36DTTGAYGA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.425051363 (573h23m2.296134408s) [noise=true , address=45.91.116.5:4323 , piece_id=7FAJKQP23A6U6SXPWVVAFVIHQA54JSPERWMCSONK5NYNHXRDLF3A, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.425681483 (573h23m2.296764528s) [noise=true , address=145.79.227.7:4303 , piece_id=MWIQ7KUZ534TFKQFMQADMBPKUFDGNFFK3JJEH3KF2CT6IQGHQRKQ, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.426480093 (573h23m2.297563138s) [noise=true , address=145.79.227.33:4877 , piece_id=IFC4ANBQU7ORCMSTYAAG3QKE4S2JPCWKDD47P5Z57V6TWAR6GFTA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.427077340 (573h23m2.298160385s) [noise=true , address=145.79.227.34:4937 , piece_id=2EYFRE3YDBIZK7YORZMWRETOEHJGDDKJVZBFW36FU6LX6PHRTDIA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.427529692 (573h23m2.298612737s) [noise=true , address=145.79.227.30:4793 , piece_id=AGUHSTJNNNRAAP5TKWIONYBZDKPJDJEQX7RX5GEGWN6TG4RELAMQ, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:02.428130125 (573h23m2.29921317s) [noise=true , address=77.90.31.16:4747 , piece_id=5TQY6C4SF6JQT6KTSEMOSY65TLPLUVCDTNVJ6U2QYFMTYP2G73VA, seg_pos={0 0}, upload=502680]: Uploading piece...
Dec 29 12:52:03.783583605 (573h23m3.65466665s) [upload=502680]: Next returned nil segment
Dec 29 12:52:03.783597080 (573h23m3.654680125s) [upload=502680]: Waiting for error group managing segments...
Dec 29 12:52:04.132513046 (573h23m4.003596091s) [noise=true , address=64.49.13.137:4494 , piece_id=K4TIPJCGNFYIMSQKXQAWABJLIHHTAS3F36I723BHTJ7YG4HUKIIQ, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.271420967 (573h23m4.142504012s) [noise=true , address=145.79.227.7:4303 , piece_id=MWIQ7KUZ534TFKQFMQADMBPKUFDGNFFK3JJEH3KF2CT6IQGHQRKQ, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.275668766 (573h23m4.146751811s) [noise=true , address=145.79.227.48:4334 , piece_id=MFEYVH6L2DBOS7Z444QFOQDVQCQS5MG3M32335L7ZCN4MXVH5L2A, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.298256927 (573h23m4.169339972s) [noise=true , address=145.79.227.17:4980 , piece_id=D3CAP7TOKGQ2NVULNTN3WSB2OQKOLVWN7TFGLPOLLJPPMHHZR7NQ, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.487163872 (573h23m4.358246917s) [noise=true , address=77.90.31.19:4863 , piece_id=XLK3SHF2GNV625H7RC5SGL67TKKBAJREGPTBCEJDX3QGD6H63GMA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.563458298 (573h23m4.434541343s) [noise=true , address=45.91.116.5:4323 , piece_id=7FAJKQP23A6U6SXPWVVAFVIHQA54JSPERWMCSONK5NYNHXRDLF3A, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.572838760 (573h23m4.443921795s) [noise=true , address=77.90.31.10:4545 , piece_id=NE3YEJJJKKOQSM2JKO5PFTPJH7SG65UZGR3QJP5JZXOUF3FN32YQ, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.580866937 (573h23m4.451949972s) [noise=true , address=77.90.31.14:4676 , piece_id=RGKFUGBFBMHXICXNPPCFXNFHOZ5EXIFPT54PBT2EA5AS2ZC5OTGA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:04.595018524 (573h23m4.466101569s) [noise=true , address=77.90.31.17:4799 , piece_id=DFRKPKCVCIPZZJB6QDU64E5JGJIIKXDEO5MJAA4X5V6N3ZTEGTFA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:05.183831178 (573h23m5.054914213s) [noise=true , address=145.79.227.30:4793 , piece_id=AGUHSTJNNNRAAP5TKWIONYBZDKPJDJEQX7RX5GEGWN6TG4RELAMQ, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:05.242293548 (573h23m5.113376593s) [noise=true , address=145.79.227.33:4877 , piece_id=IFC4ANBQU7ORCMSTYAAG3QKE4S2JPCWKDD47P5Z57V6TWAR6GFTA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:05.255836156 (573h23m5.126919201s) [noise=true , address=145.79.227.34:4937 , piece_id=2EYFRE3YDBIZK7YORZMWRETOEHJGDDKJVZBFW36FU6LX6PHRTDIA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:05.305203848 (573h23m5.176286883s) [noise=true , address=145.79.227.40:4058 , piece_id=EN2T5WMAEV3PVNAOWLMN4BUIJSAXRMAIQBQ6PKQ7PWPATZLYYBIQ, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:05.326720429 (573h23m5.197803474s) [noise=true , address=145.79.227.43:4166 , piece_id=QPEP6OT7WAFMRNNI6XVL25H3FAJ76FCQXVSEACOTV6XCYIZZSVPA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: <nil>
Dec 29 12:52:05.326747741 (573h23m5.197830786s) [seg_pos={0 0}, upload=502680]: Segment reached optimal threshold of 14 pieces.
Dec 29 12:52:05.327021833 (573h23m5.198104889s) [noise=true , address=77.90.31.18:4829 , piece_id=RG37QNRSJFFED3XOPAEOHGJNJ45MEMUON3R3MB5ASEXS4A4EQ42Q, seg_pos={0 0}, upload=502680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo): protocol: expected piece hash; context canceled; EOF
Dec 29 12:52:05.327031712 (573h23m5.198114757s) [noise=true , address=77.90.31.18:4829 , piece_id=RG37QNRSJFFED3XOPAEOHGJNJ45MEMUON3R3MB5ASEXS4A4EQ42Q, seg_pos={0 0}, upload=502680]: Long tail cancellation due to optimal threshold
Dec 29 12:52:05.327058794 (573h23m5.198141849s) [noise=true , address=64.49.13.169:4708 , piece_id=5ORIIBIW2GPJRDD5C7IBQDPKPS5SJBTSVR6SGC6NKX36DTTGAYGA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:40464->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:40464->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40464->64.49.13.169:4708: use of closed network connection
Dec 29 12:52:05.327066769 (573h23m5.198149814s) [noise=true , address=64.49.13.169:4708 , piece_id=5ORIIBIW2GPJRDD5C7IBQDPKPS5SJBTSVR6SGC6NKX36DTTGAYGA, seg_pos={0 0}, upload=502680]: Long tail cancellation due to optimal threshold
Dec 29 12:52:05.327203149 (573h23m5.198286204s) [noise=true , address=77.90.31.16:4747 , piece_id=5TQY6C4SF6JQT6KTSEMOSY65TLPLUVCDTNVJ6U2QYFMTYP2G73VA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2): protocol: expected piece hash; context canceled; EOF
Dec 29 12:52:05.327213137 (573h23m5.198296182s) [noise=true , address=77.90.31.16:4747 , piece_id=5TQY6C4SF6JQT6KTSEMOSY65TLPLUVCDTNVJ6U2QYFMTYP2G73VA, seg_pos={0 0}, upload=502680]: Long tail cancellation due to optimal threshold
Dec 29 12:52:05.327491227 (573h23m5.198574282s) [noise=true , address=193.149.164.143:4697 , piece_id=C2JF73NKW64HNXGS5BJTVZRCAIVWKZBGDGUZFJMDUWW3RBBXJDSA, seg_pos={0 0}, upload=502680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF
Dec 29 12:52:05.327500265 (573h23m5.1985833s) [noise=true , address=193.149.164.143:4697 , piece_id=C2JF73NKW64HNXGS5BJTVZRCAIVWKZBGDGUZFJMDUWW3RBBXJDSA, seg_pos={0 0}, upload=502680]: Long tail cancellation due to optimal threshold
Dec 29 12:52:05.327548276 (573h23m5.198631321s) [seg_pos={0 0}, upload=502680]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 12:52:05.327736143 (573h23m5.198819188s) [upload=502680]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 12:52:05.327744139 (573h23m5.198827174s) [upload=502680]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 12:52:05.624748327 (573h23m5.495831362s) [upload=502680]: Done uploading segments...
Dec 29 12:52:05.624772935 (573h23m5.49585597s) [upload=502680]: Upload finished. err: <nil>
Duration: 50.065647905s
core
uplink
us-select-1
us1-placement-20
|