Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X..X.....X.X.....X...XX.X......XXX.XX.X.XX.......X..XXXXXX..] duration: 5.686153433s trace: https://jaeger.dev.storj.io/trace/8865108ac6c41cd FAIL (2025-12-29 18:48:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: N3XZ42R2Q4L4DWF2TTXUZLYO6GHNDNHVYZA3OHUMK6C7THTYYJOQ) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: LE64NY6GYIO5C4YOZCVA4WYKTGVSH7YADNE7IPYECFXJOACJKN4Q) context deadline exceeded ======LOGS====== Dec 29 18:46:42.441205837 (579h17m42.312288872s) [upload=507821]: Starting upload Dec 29 18:46:42.441261373 (579h17m42.312344408s) [upload=507821]: Done starting upload Dec 29 18:46:42.441411128 (579h17m42.312494163s) [upload=507821]: Uploading segments... Dec 29 18:46:42.441497002 (579h17m42.312580037s) [upload=507821]: Got next segment. Inline: false Dec 29 18:46:42.441529704 (579h17m42.312612739s) [upload=507821]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:46:42.441534223 (579h17m42.312617268s) [upload=507821]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:46:42.683954376 (579h17m42.555037442s) [seg_pos={0 0}, upload=507821]: Begin upload segment... Dec 29 18:46:42.684023749 (579h17m42.555106794s) [seg_pos={0 0}, upload=507821]: Done begin upload segment. Dec 29 18:46:42.684075667 (579h17m42.555158712s) [noise=true , address=45.91.116.5:4297 , piece_id=N3XZ42R2Q4L4DWF2TTXUZLYO6GHNDNHVYZA3OHUMK6C7THTYYJOQ, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.684213550 (579h17m42.555296625s) [noise=true , address=145.79.227.15:4876 , piece_id=OWWJPFOABX2H7OHHSEOT3CFJTSD6OJYENZH5XZGITSSGWBKTS2YA, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.684646804 (579h17m42.555729839s) [noise=true , address=45.91.116.4:4939 , piece_id=7UFGBF2AY7P7J3EIZLUGE3IBRT7PTQ5IWQE53E7YBJBIVNMNZD5A, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.684928071 (579h17m42.556011106s) [noise=true , address=145.79.227.36:4518 , piece_id=X2JN2AEWBV22TNOO6UKRO7XNAMQSV5JNZV6ZZ2OCNZ4CFKYV4PSQ, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.685202202 (579h17m42.556285247s) [noise=true , address=145.79.227.32:4835 , piece_id=3CRXHKTSHCG7CE34GUNMXQKEV3QVYKUJ3SY3HFY7Z4376522HW7A, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.685302613 (579h17m42.556385678s) [noise=true , address=145.79.227.50:4373 , piece_id=R37T4QAGDOOUHANM2KCM55IA25AYEPBZWRNXQ3CPHVP7GOR7XN6A, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.685468539 (579h17m42.556551584s) [noise=true , address=145.79.227.11:4730 , piece_id=OGLKWCPU6WCB7WBHOJRTN2IMTOBA5J5D2YF4V7KMZB5DUNQGSJMA, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.685718826 (579h17m42.556801861s) [noise=true , address=145.79.227.41:4086 , piece_id=G43WMWOOAMMSD7Z6EMM5UAWLMBFZUW6FQQP434RNLI6ZBP47A4GQ, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.686019589 (579h17m42.557102644s) [noise=true , address=145.79.227.44:4194 , piece_id=ERT3ZSY3SAVBD6H45ULEKTCQ3YLIGSNGGYJ3O3IOI4NC5XCFDJSA, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.686040379 (579h17m42.557123414s) [noise=true , address=145.79.227.19:4079 , piece_id=2NQM2EDPKF47YW3DRWA4XH2XMRIPM5LXBOZXHS33GRPC3EM2N7BA, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.686388171 (579h17m42.557471206s) [noise=true , address=77.90.31.21:4878 , piece_id=LE64NY6GYIO5C4YOZCVA4WYKTGVSH7YADNE7IPYECFXJOACJKN4Q, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.686823189 (579h17m42.557906244s) [noise=true , address=45.91.116.13:4155 , piece_id=JP6FIWK6ZUL7JYCJZP4MIWSAO7JKJYMGYBO4TYPBZZWS5OA3D32A, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.687546166 (579h17m42.558629221s) [noise=true , address=77.90.31.14:4664 , piece_id=IGVA5XK32MIUFO4VCLUJQFXY22QWZGPWJGC3GFKDBGOWUOUEUSDA, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.687785712 (579h17m42.558868747s) [noise=true , address=145.79.227.5:4221 , piece_id=BN2SPT5A3QSA5652XBRER64MOQGOAXMJIDHVZIVOSHLV66U44UQA, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.688122132 (579h17m42.559205167s) [noise=true , address=77.90.31.18:4805 , piece_id=QR6O3XMAKUFZIWORLW3WSVSUFKZPRXV6IN4CTJI7LMNCEBBKEQPQ, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.688422364 (579h17m42.559505399s) [noise=true , address=145.79.227.37:4565 , piece_id=RFNRTNLRK7WWHSZ4OGJZ7EKFDZMK53JGDORFPYLMMMIGK67IAVNQ, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:44.374325810 (579h17m44.245408845s) [upload=507821]: Next returned nil segment Dec 29 18:46:44.374344285 (579h17m44.24542733s) [upload=507821]: Waiting for error group managing segments... Dec 29 18:46:44.663336120 (579h17m44.534419155s) [noise=true , address=145.79.227.50:4373 , piece_id=R37T4QAGDOOUHANM2KCM55IA25AYEPBZWRNXQ3CPHVP7GOR7XN6A, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.663972081 (579h17m44.535055116s) [noise=true , address=145.79.227.5:4221 , piece_id=BN2SPT5A3QSA5652XBRER64MOQGOAXMJIDHVZIVOSHLV66U44UQA, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.669270359 (579h17m44.540353424s) [noise=true , address=145.79.227.19:4079 , piece_id=2NQM2EDPKF47YW3DRWA4XH2XMRIPM5LXBOZXHS33GRPC3EM2N7BA, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.680008927 (579h17m44.551091972s) [noise=true , address=145.79.227.32:4835 , piece_id=3CRXHKTSHCG7CE34GUNMXQKEV3QVYKUJ3SY3HFY7Z4376522HW7A, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.683014760 (579h17m44.554097795s) [noise=true , address=145.79.227.41:4086 , piece_id=G43WMWOOAMMSD7Z6EMM5UAWLMBFZUW6FQQP434RNLI6ZBP47A4GQ, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.684838594 (579h17m44.555921629s) [noise=true , address=145.79.227.37:4565 , piece_id=RFNRTNLRK7WWHSZ4OGJZ7EKFDZMK53JGDORFPYLMMMIGK67IAVNQ, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.686348458 (579h17m44.557431503s) [noise=true , address=145.79.227.15:4876 , piece_id=OWWJPFOABX2H7OHHSEOT3CFJTSD6OJYENZH5XZGITSSGWBKTS2YA, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.689968521 (579h17m44.561051557s) [noise=true , address=145.79.227.36:4518 , piece_id=X2JN2AEWBV22TNOO6UKRO7XNAMQSV5JNZV6ZZ2OCNZ4CFKYV4PSQ, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.694047639 (579h17m44.565130674s) [noise=true , address=145.79.227.11:4730 , piece_id=OGLKWCPU6WCB7WBHOJRTN2IMTOBA5J5D2YF4V7KMZB5DUNQGSJMA, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.857023028 (579h17m44.728106063s) [noise=true , address=77.90.31.14:4664 , piece_id=IGVA5XK32MIUFO4VCLUJQFXY22QWZGPWJGC3GFKDBGOWUOUEUSDA, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.858769754 (579h17m44.729852789s) [noise=true , address=77.90.31.21:4878 , piece_id=LE64NY6GYIO5C4YOZCVA4WYKTGVSH7YADNE7IPYECFXJOACJKN4Q, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.917550793 (579h17m44.788633828s) [noise=true , address=77.90.31.18:4805 , piece_id=QR6O3XMAKUFZIWORLW3WSVSUFKZPRXV6IN4CTJI7LMNCEBBKEQPQ, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.921883543 (579h17m44.792966578s) [noise=true , address=45.91.116.5:4297 , piece_id=N3XZ42R2Q4L4DWF2TTXUZLYO6GHNDNHVYZA3OHUMK6C7THTYYJOQ, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.921900045 (579h17m44.79298308s) [seg_pos={0 0}, upload=507821]: Segment reached optimal threshold of 13 pieces. Dec 29 18:46:44.922077011 (579h17m44.793160046s) [noise=true , address=45.91.116.13:4155 , piece_id=JP6FIWK6ZUL7JYCJZP4MIWSAO7JKJYMGYBO4TYPBZZWS5OA3D32A, seg_pos={0 0}, upload=507821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 18:46:44.922152325 (579h17m44.79323536s) [noise=true , address=45.91.116.13:4155 , piece_id=JP6FIWK6ZUL7JYCJZP4MIWSAO7JKJYMGYBO4TYPBZZWS5OA3D32A, seg_pos={0 0}, upload=507821]: Long tail cancellation due to optimal threshold Dec 29 18:46:44.922388925 (579h17m44.79347196s) [noise=true , address=45.91.116.4:4939 , piece_id=7UFGBF2AY7P7J3EIZLUGE3IBRT7PTQ5IWQE53E7YBJBIVNMNZD5A, seg_pos={0 0}, upload=507821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 29 18:46:44.922397311 (579h17m44.793480356s) [noise=true , address=45.91.116.4:4939 , piece_id=7UFGBF2AY7P7J3EIZLUGE3IBRT7PTQ5IWQE53E7YBJBIVNMNZD5A, seg_pos={0 0}, upload=507821]: Long tail cancellation due to optimal threshold Dec 29 18:46:44.922461954 (579h17m44.793544989s) [noise=true , address=145.79.227.44:4194 , piece_id=ERT3ZSY3SAVBD6H45ULEKTCQ3YLIGSNGGYJ3O3IOI4NC5XCFDJSA, seg_pos={0 0}, upload=507821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 18:46:44.922478305 (579h17m44.79356134s) [noise=true , address=145.79.227.44:4194 , piece_id=ERT3ZSY3SAVBD6H45ULEKTCQ3YLIGSNGGYJ3O3IOI4NC5XCFDJSA, seg_pos={0 0}, upload=507821]: Long tail cancellation due to optimal threshold Dec 29 18:46:44.922528621 (579h17m44.793611656s) [seg_pos={0 0}, upload=507821]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:46:44.922638531 (579h17m44.793721566s) [upload=507821]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:46:44.922644632 (579h17m44.793727667s) [upload=507821]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:46:45.214309527 (579h17m45.085392572s) [upload=507821]: Done uploading segments... Dec 29 18:46:45.214339383 (579h17m45.085422428s) [upload=507821]: Upload finished. err: <nil> Duration: 5.68617858s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XX..X.....X.X.....X...XX.X......XXX.XX.X.XX.......X..XXXXXX.] duration: 5.674427785s trace: https://jaeger.dev.storj.io/trace/8865108ac6c41cd FAIL (2025-12-29 18:48:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: N3XZ42R2Q4L4DWF2TTXUZLYO6GHNDNHVYZA3OHUMK6C7THTYYJOQ) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: LE64NY6GYIO5C4YOZCVA4WYKTGVSH7YADNE7IPYECFXJOACJKN4Q) context deadline exceeded ======LOGS====== Dec 29 18:46:42.441205837 (579h17m42.312288872s) [upload=507821]: Starting upload Dec 29 18:46:42.441261373 (579h17m42.312344408s) [upload=507821]: Done starting upload Dec 29 18:46:42.441411128 (579h17m42.312494163s) [upload=507821]: Uploading segments... Dec 29 18:46:42.441497002 (579h17m42.312580037s) [upload=507821]: Got next segment. Inline: false Dec 29 18:46:42.441529704 (579h17m42.312612739s) [upload=507821]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:46:42.441534223 (579h17m42.312617268s) [upload=507821]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:46:42.683954376 (579h17m42.555037442s) [seg_pos={0 0}, upload=507821]: Begin upload segment... Dec 29 18:46:42.684023749 (579h17m42.555106794s) [seg_pos={0 0}, upload=507821]: Done begin upload segment. Dec 29 18:46:42.684075667 (579h17m42.555158712s) [noise=true , address=45.91.116.5:4297 , piece_id=N3XZ42R2Q4L4DWF2TTXUZLYO6GHNDNHVYZA3OHUMK6C7THTYYJOQ, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.684213550 (579h17m42.555296625s) [noise=true , address=145.79.227.15:4876 , piece_id=OWWJPFOABX2H7OHHSEOT3CFJTSD6OJYENZH5XZGITSSGWBKTS2YA, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.684646804 (579h17m42.555729839s) [noise=true , address=45.91.116.4:4939 , piece_id=7UFGBF2AY7P7J3EIZLUGE3IBRT7PTQ5IWQE53E7YBJBIVNMNZD5A, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.684928071 (579h17m42.556011106s) [noise=true , address=145.79.227.36:4518 , piece_id=X2JN2AEWBV22TNOO6UKRO7XNAMQSV5JNZV6ZZ2OCNZ4CFKYV4PSQ, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.685202202 (579h17m42.556285247s) [noise=true , address=145.79.227.32:4835 , piece_id=3CRXHKTSHCG7CE34GUNMXQKEV3QVYKUJ3SY3HFY7Z4376522HW7A, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.685302613 (579h17m42.556385678s) [noise=true , address=145.79.227.50:4373 , piece_id=R37T4QAGDOOUHANM2KCM55IA25AYEPBZWRNXQ3CPHVP7GOR7XN6A, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.685468539 (579h17m42.556551584s) [noise=true , address=145.79.227.11:4730 , piece_id=OGLKWCPU6WCB7WBHOJRTN2IMTOBA5J5D2YF4V7KMZB5DUNQGSJMA, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.685718826 (579h17m42.556801861s) [noise=true , address=145.79.227.41:4086 , piece_id=G43WMWOOAMMSD7Z6EMM5UAWLMBFZUW6FQQP434RNLI6ZBP47A4GQ, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.686019589 (579h17m42.557102644s) [noise=true , address=145.79.227.44:4194 , piece_id=ERT3ZSY3SAVBD6H45ULEKTCQ3YLIGSNGGYJ3O3IOI4NC5XCFDJSA, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.686040379 (579h17m42.557123414s) [noise=true , address=145.79.227.19:4079 , piece_id=2NQM2EDPKF47YW3DRWA4XH2XMRIPM5LXBOZXHS33GRPC3EM2N7BA, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.686388171 (579h17m42.557471206s) [noise=true , address=77.90.31.21:4878 , piece_id=LE64NY6GYIO5C4YOZCVA4WYKTGVSH7YADNE7IPYECFXJOACJKN4Q, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.686823189 (579h17m42.557906244s) [noise=true , address=45.91.116.13:4155 , piece_id=JP6FIWK6ZUL7JYCJZP4MIWSAO7JKJYMGYBO4TYPBZZWS5OA3D32A, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.687546166 (579h17m42.558629221s) [noise=true , address=77.90.31.14:4664 , piece_id=IGVA5XK32MIUFO4VCLUJQFXY22QWZGPWJGC3GFKDBGOWUOUEUSDA, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.687785712 (579h17m42.558868747s) [noise=true , address=145.79.227.5:4221 , piece_id=BN2SPT5A3QSA5652XBRER64MOQGOAXMJIDHVZIVOSHLV66U44UQA, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.688122132 (579h17m42.559205167s) [noise=true , address=77.90.31.18:4805 , piece_id=QR6O3XMAKUFZIWORLW3WSVSUFKZPRXV6IN4CTJI7LMNCEBBKEQPQ, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.688422364 (579h17m42.559505399s) [noise=true , address=145.79.227.37:4565 , piece_id=RFNRTNLRK7WWHSZ4OGJZ7EKFDZMK53JGDORFPYLMMMIGK67IAVNQ, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:44.374325810 (579h17m44.245408845s) [upload=507821]: Next returned nil segment Dec 29 18:46:44.374344285 (579h17m44.24542733s) [upload=507821]: Waiting for error group managing segments... Dec 29 18:46:44.663336120 (579h17m44.534419155s) [noise=true , address=145.79.227.50:4373 , piece_id=R37T4QAGDOOUHANM2KCM55IA25AYEPBZWRNXQ3CPHVP7GOR7XN6A, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.663972081 (579h17m44.535055116s) [noise=true , address=145.79.227.5:4221 , piece_id=BN2SPT5A3QSA5652XBRER64MOQGOAXMJIDHVZIVOSHLV66U44UQA, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.669270359 (579h17m44.540353424s) [noise=true , address=145.79.227.19:4079 , piece_id=2NQM2EDPKF47YW3DRWA4XH2XMRIPM5LXBOZXHS33GRPC3EM2N7BA, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.680008927 (579h17m44.551091972s) [noise=true , address=145.79.227.32:4835 , piece_id=3CRXHKTSHCG7CE34GUNMXQKEV3QVYKUJ3SY3HFY7Z4376522HW7A, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.683014760 (579h17m44.554097795s) [noise=true , address=145.79.227.41:4086 , piece_id=G43WMWOOAMMSD7Z6EMM5UAWLMBFZUW6FQQP434RNLI6ZBP47A4GQ, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.684838594 (579h17m44.555921629s) [noise=true , address=145.79.227.37:4565 , piece_id=RFNRTNLRK7WWHSZ4OGJZ7EKFDZMK53JGDORFPYLMMMIGK67IAVNQ, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.686348458 (579h17m44.557431503s) [noise=true , address=145.79.227.15:4876 , piece_id=OWWJPFOABX2H7OHHSEOT3CFJTSD6OJYENZH5XZGITSSGWBKTS2YA, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.689968521 (579h17m44.561051557s) [noise=true , address=145.79.227.36:4518 , piece_id=X2JN2AEWBV22TNOO6UKRO7XNAMQSV5JNZV6ZZ2OCNZ4CFKYV4PSQ, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.694047639 (579h17m44.565130674s) [noise=true , address=145.79.227.11:4730 , piece_id=OGLKWCPU6WCB7WBHOJRTN2IMTOBA5J5D2YF4V7KMZB5DUNQGSJMA, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.857023028 (579h17m44.728106063s) [noise=true , address=77.90.31.14:4664 , piece_id=IGVA5XK32MIUFO4VCLUJQFXY22QWZGPWJGC3GFKDBGOWUOUEUSDA, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.858769754 (579h17m44.729852789s) [noise=true , address=77.90.31.21:4878 , piece_id=LE64NY6GYIO5C4YOZCVA4WYKTGVSH7YADNE7IPYECFXJOACJKN4Q, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.917550793 (579h17m44.788633828s) [noise=true , address=77.90.31.18:4805 , piece_id=QR6O3XMAKUFZIWORLW3WSVSUFKZPRXV6IN4CTJI7LMNCEBBKEQPQ, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.921883543 (579h17m44.792966578s) [noise=true , address=45.91.116.5:4297 , piece_id=N3XZ42R2Q4L4DWF2TTXUZLYO6GHNDNHVYZA3OHUMK6C7THTYYJOQ, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.921900045 (579h17m44.79298308s) [seg_pos={0 0}, upload=507821]: Segment reached optimal threshold of 13 pieces. Dec 29 18:46:44.922077011 (579h17m44.793160046s) [noise=true , address=45.91.116.13:4155 , piece_id=JP6FIWK6ZUL7JYCJZP4MIWSAO7JKJYMGYBO4TYPBZZWS5OA3D32A, seg_pos={0 0}, upload=507821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 18:46:44.922152325 (579h17m44.79323536s) [noise=true , address=45.91.116.13:4155 , piece_id=JP6FIWK6ZUL7JYCJZP4MIWSAO7JKJYMGYBO4TYPBZZWS5OA3D32A, seg_pos={0 0}, upload=507821]: Long tail cancellation due to optimal threshold Dec 29 18:46:44.922388925 (579h17m44.79347196s) [noise=true , address=45.91.116.4:4939 , piece_id=7UFGBF2AY7P7J3EIZLUGE3IBRT7PTQ5IWQE53E7YBJBIVNMNZD5A, seg_pos={0 0}, upload=507821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 29 18:46:44.922397311 (579h17m44.793480356s) [noise=true , address=45.91.116.4:4939 , piece_id=7UFGBF2AY7P7J3EIZLUGE3IBRT7PTQ5IWQE53E7YBJBIVNMNZD5A, seg_pos={0 0}, upload=507821]: Long tail cancellation due to optimal threshold Dec 29 18:46:44.922461954 (579h17m44.793544989s) [noise=true , address=145.79.227.44:4194 , piece_id=ERT3ZSY3SAVBD6H45ULEKTCQ3YLIGSNGGYJ3O3IOI4NC5XCFDJSA, seg_pos={0 0}, upload=507821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 18:46:44.922478305 (579h17m44.79356134s) [noise=true , address=145.79.227.44:4194 , piece_id=ERT3ZSY3SAVBD6H45ULEKTCQ3YLIGSNGGYJ3O3IOI4NC5XCFDJSA, seg_pos={0 0}, upload=507821]: Long tail cancellation due to optimal threshold Dec 29 18:46:44.922528621 (579h17m44.793611656s) [seg_pos={0 0}, upload=507821]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:46:44.922638531 (579h17m44.793721566s) [upload=507821]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:46:44.922644632 (579h17m44.793727667s) [upload=507821]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:46:45.214309527 (579h17m45.085392572s) [upload=507821]: Done uploading segments... Dec 29 18:46:45.214339383 (579h17m45.085422428s) [upload=507821]: Upload finished. err: <nil> Duration: 5.674456852s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 23.307545436s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:47:27 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 23.307584201s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XXX..X.....X.X.....X...XX.X......XXX.XX.X.XX.......X..XXXXXX] duration: 30.002468845s trace: https://jaeger.dev.storj.io/trace/8865108ac6c41cd FAIL (2025-12-29 18:47:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: N3XZ42R2Q4L4DWF2TTXUZLYO6GHNDNHVYZA3OHUMK6C7THTYYJOQ) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: LE64NY6GYIO5C4YOZCVA4WYKTGVSH7YADNE7IPYECFXJOACJKN4Q) context deadline exceeded ======LOGS====== Dec 29 18:46:42.441205837 (579h17m42.312288872s) [upload=507821]: Starting upload Dec 29 18:46:42.441261373 (579h17m42.312344408s) [upload=507821]: Done starting upload Dec 29 18:46:42.441411128 (579h17m42.312494163s) [upload=507821]: Uploading segments... Dec 29 18:46:42.441497002 (579h17m42.312580037s) [upload=507821]: Got next segment. Inline: false Dec 29 18:46:42.441529704 (579h17m42.312612739s) [upload=507821]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:46:42.441534223 (579h17m42.312617268s) [upload=507821]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:46:42.683954376 (579h17m42.555037442s) [seg_pos={0 0}, upload=507821]: Begin upload segment... Dec 29 18:46:42.684023749 (579h17m42.555106794s) [seg_pos={0 0}, upload=507821]: Done begin upload segment. Dec 29 18:46:42.684075667 (579h17m42.555158712s) [noise=true , address=45.91.116.5:4297 , piece_id=N3XZ42R2Q4L4DWF2TTXUZLYO6GHNDNHVYZA3OHUMK6C7THTYYJOQ, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.684213550 (579h17m42.555296625s) [noise=true , address=145.79.227.15:4876 , piece_id=OWWJPFOABX2H7OHHSEOT3CFJTSD6OJYENZH5XZGITSSGWBKTS2YA, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.684646804 (579h17m42.555729839s) [noise=true , address=45.91.116.4:4939 , piece_id=7UFGBF2AY7P7J3EIZLUGE3IBRT7PTQ5IWQE53E7YBJBIVNMNZD5A, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.684928071 (579h17m42.556011106s) [noise=true , address=145.79.227.36:4518 , piece_id=X2JN2AEWBV22TNOO6UKRO7XNAMQSV5JNZV6ZZ2OCNZ4CFKYV4PSQ, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.685202202 (579h17m42.556285247s) [noise=true , address=145.79.227.32:4835 , piece_id=3CRXHKTSHCG7CE34GUNMXQKEV3QVYKUJ3SY3HFY7Z4376522HW7A, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.685302613 (579h17m42.556385678s) [noise=true , address=145.79.227.50:4373 , piece_id=R37T4QAGDOOUHANM2KCM55IA25AYEPBZWRNXQ3CPHVP7GOR7XN6A, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.685468539 (579h17m42.556551584s) [noise=true , address=145.79.227.11:4730 , piece_id=OGLKWCPU6WCB7WBHOJRTN2IMTOBA5J5D2YF4V7KMZB5DUNQGSJMA, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.685718826 (579h17m42.556801861s) [noise=true , address=145.79.227.41:4086 , piece_id=G43WMWOOAMMSD7Z6EMM5UAWLMBFZUW6FQQP434RNLI6ZBP47A4GQ, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.686019589 (579h17m42.557102644s) [noise=true , address=145.79.227.44:4194 , piece_id=ERT3ZSY3SAVBD6H45ULEKTCQ3YLIGSNGGYJ3O3IOI4NC5XCFDJSA, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.686040379 (579h17m42.557123414s) [noise=true , address=145.79.227.19:4079 , piece_id=2NQM2EDPKF47YW3DRWA4XH2XMRIPM5LXBOZXHS33GRPC3EM2N7BA, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.686388171 (579h17m42.557471206s) [noise=true , address=77.90.31.21:4878 , piece_id=LE64NY6GYIO5C4YOZCVA4WYKTGVSH7YADNE7IPYECFXJOACJKN4Q, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.686823189 (579h17m42.557906244s) [noise=true , address=45.91.116.13:4155 , piece_id=JP6FIWK6ZUL7JYCJZP4MIWSAO7JKJYMGYBO4TYPBZZWS5OA3D32A, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.687546166 (579h17m42.558629221s) [noise=true , address=77.90.31.14:4664 , piece_id=IGVA5XK32MIUFO4VCLUJQFXY22QWZGPWJGC3GFKDBGOWUOUEUSDA, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.687785712 (579h17m42.558868747s) [noise=true , address=145.79.227.5:4221 , piece_id=BN2SPT5A3QSA5652XBRER64MOQGOAXMJIDHVZIVOSHLV66U44UQA, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.688122132 (579h17m42.559205167s) [noise=true , address=77.90.31.18:4805 , piece_id=QR6O3XMAKUFZIWORLW3WSVSUFKZPRXV6IN4CTJI7LMNCEBBKEQPQ, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:42.688422364 (579h17m42.559505399s) [noise=true , address=145.79.227.37:4565 , piece_id=RFNRTNLRK7WWHSZ4OGJZ7EKFDZMK53JGDORFPYLMMMIGK67IAVNQ, seg_pos={0 0}, upload=507821]: Uploading piece... Dec 29 18:46:44.374325810 (579h17m44.245408845s) [upload=507821]: Next returned nil segment Dec 29 18:46:44.374344285 (579h17m44.24542733s) [upload=507821]: Waiting for error group managing segments... Dec 29 18:46:44.663336120 (579h17m44.534419155s) [noise=true , address=145.79.227.50:4373 , piece_id=R37T4QAGDOOUHANM2KCM55IA25AYEPBZWRNXQ3CPHVP7GOR7XN6A, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.663972081 (579h17m44.535055116s) [noise=true , address=145.79.227.5:4221 , piece_id=BN2SPT5A3QSA5652XBRER64MOQGOAXMJIDHVZIVOSHLV66U44UQA, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.669270359 (579h17m44.540353424s) [noise=true , address=145.79.227.19:4079 , piece_id=2NQM2EDPKF47YW3DRWA4XH2XMRIPM5LXBOZXHS33GRPC3EM2N7BA, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.680008927 (579h17m44.551091972s) [noise=true , address=145.79.227.32:4835 , piece_id=3CRXHKTSHCG7CE34GUNMXQKEV3QVYKUJ3SY3HFY7Z4376522HW7A, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.683014760 (579h17m44.554097795s) [noise=true , address=145.79.227.41:4086 , piece_id=G43WMWOOAMMSD7Z6EMM5UAWLMBFZUW6FQQP434RNLI6ZBP47A4GQ, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.684838594 (579h17m44.555921629s) [noise=true , address=145.79.227.37:4565 , piece_id=RFNRTNLRK7WWHSZ4OGJZ7EKFDZMK53JGDORFPYLMMMIGK67IAVNQ, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.686348458 (579h17m44.557431503s) [noise=true , address=145.79.227.15:4876 , piece_id=OWWJPFOABX2H7OHHSEOT3CFJTSD6OJYENZH5XZGITSSGWBKTS2YA, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.689968521 (579h17m44.561051557s) [noise=true , address=145.79.227.36:4518 , piece_id=X2JN2AEWBV22TNOO6UKRO7XNAMQSV5JNZV6ZZ2OCNZ4CFKYV4PSQ, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.694047639 (579h17m44.565130674s) [noise=true , address=145.79.227.11:4730 , piece_id=OGLKWCPU6WCB7WBHOJRTN2IMTOBA5J5D2YF4V7KMZB5DUNQGSJMA, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.857023028 (579h17m44.728106063s) [noise=true , address=77.90.31.14:4664 , piece_id=IGVA5XK32MIUFO4VCLUJQFXY22QWZGPWJGC3GFKDBGOWUOUEUSDA, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.858769754 (579h17m44.729852789s) [noise=true , address=77.90.31.21:4878 , piece_id=LE64NY6GYIO5C4YOZCVA4WYKTGVSH7YADNE7IPYECFXJOACJKN4Q, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.917550793 (579h17m44.788633828s) [noise=true , address=77.90.31.18:4805 , piece_id=QR6O3XMAKUFZIWORLW3WSVSUFKZPRXV6IN4CTJI7LMNCEBBKEQPQ, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.921883543 (579h17m44.792966578s) [noise=true , address=45.91.116.5:4297 , piece_id=N3XZ42R2Q4L4DWF2TTXUZLYO6GHNDNHVYZA3OHUMK6C7THTYYJOQ, seg_pos={0 0}, upload=507821]: Done uploading piece. err: <nil> Dec 29 18:46:44.921900045 (579h17m44.79298308s) [seg_pos={0 0}, upload=507821]: Segment reached optimal threshold of 13 pieces. Dec 29 18:46:44.922077011 (579h17m44.793160046s) [noise=true , address=45.91.116.13:4155 , piece_id=JP6FIWK6ZUL7JYCJZP4MIWSAO7JKJYMGYBO4TYPBZZWS5OA3D32A, seg_pos={0 0}, upload=507821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 29 18:46:44.922152325 (579h17m44.79323536s) [noise=true , address=45.91.116.13:4155 , piece_id=JP6FIWK6ZUL7JYCJZP4MIWSAO7JKJYMGYBO4TYPBZZWS5OA3D32A, seg_pos={0 0}, upload=507821]: Long tail cancellation due to optimal threshold Dec 29 18:46:44.922388925 (579h17m44.79347196s) [noise=true , address=45.91.116.4:4939 , piece_id=7UFGBF2AY7P7J3EIZLUGE3IBRT7PTQ5IWQE53E7YBJBIVNMNZD5A, seg_pos={0 0}, upload=507821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 29 18:46:44.922397311 (579h17m44.793480356s) [noise=true , address=45.91.116.4:4939 , piece_id=7UFGBF2AY7P7J3EIZLUGE3IBRT7PTQ5IWQE53E7YBJBIVNMNZD5A, seg_pos={0 0}, upload=507821]: Long tail cancellation due to optimal threshold Dec 29 18:46:44.922461954 (579h17m44.793544989s) [noise=true , address=145.79.227.44:4194 , piece_id=ERT3ZSY3SAVBD6H45ULEKTCQ3YLIGSNGGYJ3O3IOI4NC5XCFDJSA, seg_pos={0 0}, upload=507821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 18:46:44.922478305 (579h17m44.79356134s) [noise=true , address=145.79.227.44:4194 , piece_id=ERT3ZSY3SAVBD6H45ULEKTCQ3YLIGSNGGYJ3O3IOI4NC5XCFDJSA, seg_pos={0 0}, upload=507821]: Long tail cancellation due to optimal threshold Dec 29 18:46:44.922528621 (579h17m44.793611656s) [seg_pos={0 0}, upload=507821]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:46:44.922638531 (579h17m44.793721566s) [upload=507821]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:46:44.922644632 (579h17m44.793727667s) [upload=507821]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:46:45.214309527 (579h17m45.085392572s) [upload=507821]: Done uploading segments... Dec 29 18:46:45.214339383 (579h17m45.085422428s) [upload=507821]: Upload finished. err: <nil> Duration: 30.002494114s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 15.473079618s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:47:01 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 15.473113373s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 15.537370703s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:46:17 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 15.537397305s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.XXX..X.....X.X.....X...XX.X......XXX.XX.X.XX.......X..XXXXX] duration: 30.004588709s trace: https://jaeger.dev.storj.io/trace/18a2ca15f8ebbad5 FAIL (2025-12-29 18:45:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: DPYMF7AEO7E6LDMI7MM27F7PAVWUWP3H3J55KRC7VGCWHEXULPTQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: SHBEQUDUNTS2VSBU7WSBNGDVV5SCENXGC3BLRIG32W4SCN5DEWSA) context deadline exceeded; piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: R2MWDEFNPMI63DXBICR3OINB7L3MYNPTXVNG7HOMILXOHPCJYMWQ) context deadline exceeded ======LOGS====== Dec 29 18:45:29.997405538 (579h16m29.868488583s) [upload=507801]: Starting upload Dec 29 18:45:29.997461184 (579h16m29.868544219s) [upload=507801]: Done starting upload Dec 29 18:45:29.997681694 (579h16m29.868764739s) [upload=507801]: Uploading segments... Dec 29 18:45:29.997721651 (579h16m29.868804686s) [upload=507801]: Got next segment. Inline: false Dec 29 18:45:29.997747018 (579h16m29.868830053s) [upload=507801]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:45:29.997751326 (579h16m29.868834361s) [upload=507801]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:45:30.246555027 (579h16m30.117638062s) [seg_pos={0 0}, upload=507801]: Begin upload segment... Dec 29 18:45:30.246595785 (579h16m30.11767883s) [seg_pos={0 0}, upload=507801]: Done begin upload segment. Dec 29 18:45:30.246618116 (579h16m30.117701152s) [noise=true , address=77.90.31.12:4589 , piece_id=DPYMF7AEO7E6LDMI7MM27F7PAVWUWP3H3J55KRC7VGCWHEXULPTQ, seg_pos={0 0}, upload=507801]: Uploading piece... Dec 29 18:45:30.246684263 (579h16m30.117767298s) [noise=true , address=145.79.227.12:4765 , piece_id=EXHDKYB3KSQMBDXSX46EBQPBZUGY5FYJ22G2YTGO2W66DISTLD4A, seg_pos={0 0}, upload=507801]: Uploading piece... Dec 29 18:45:30.247372333 (579h16m30.118455368s) [noise=true , address=45.91.116.14:4192 , piece_id=SHBEQUDUNTS2VSBU7WSBNGDVV5SCENXGC3BLRIG32W4SCN5DEWSA, seg_pos={0 0}, upload=507801]: Uploading piece... Dec 29 18:45:30.247965813 (579h16m30.119048848s) [noise=true , address=145.79.227.5:4222 , piece_id=UC5LM3TSFWCW6JQOLYK5773UKU67CAN2AGEA4GMIPSPH6FI4VOXA, seg_pos={0 0}, upload=507801]: Uploading piece... Dec 29 18:45:30.248338423 (579h16m30.119421468s) [noise=true , address=45.91.116.12:4225 , piece_id=R2MWDEFNPMI63DXBICR3OINB7L3MYNPTXVNG7HOMILXOHPCJYMWQ, seg_pos={0 0}, upload=507801]: Uploading piece... Dec 29 18:45:30.248689090 (579h16m30.119772136s) [noise=true , address=145.79.227.11:4729 , piece_id=SHSSR2UTR4Z5QOPABQB2SYBA3JLQME6XCKKVEGTDHGQHY7R4Y62A, seg_pos={0 0}, upload=507801]: Uploading piece... Dec 29 18:45:30.249059857 (579h16m30.120142892s) [noise=true , address=45.91.116.4:4939 , piece_id=BKAMYPCAXF24VHJRC5UAKPCA4XOUFBKMVOWFCFVD2VIJLPVKCKDQ, seg_pos={0 0}, upload=507801]: Uploading piece... Dec 29 18:45:30.249534961 (579h16m30.120618006s) [noise=true , address=145.79.227.49:4339 , piece_id=B3UJYKAMRVRWBDBDTU47GJBOC7VKZ377DQW363OKIJPT5Y6YUPJA, seg_pos={0 0}, upload=507801]: Uploading piece... Dec 29 18:45:30.250712433 (579h16m30.121795468s) [noise=true , address=45.91.116.6:4973 , piece_id=OR335JA6AUPEKUM6FV5OPYPCG5AH55DPCZDDS5VMTY6BRNM6H5EQ, seg_pos={0 0}, upload=507801]: Uploading piece... Dec 29 18:45:30.252240303 (579h16m30.123323338s) [noise=true , address=145.79.227.4:4186 , piece_id=VLYTHEBWPLMEAUF5LFJ4XXS4KMQDXHZK4TMAP4BETZF4DDGVFEXA, seg_pos={0 0}, upload=507801]: Uploading piece... Dec 29 18:45:30.254282380 (579h16m30.125365415s) [noise=true , address=145.79.227.45:4229 , piece_id=HJNSE7MRTSLQWGSQCYLUCBSEDFEV3K64EB4DERKUXHLJYDR3UHIQ, seg_pos={0 0}, upload=507801]: Uploading piece... Dec 29 18:45:30.258952543 (579h16m30.130035588s) [noise=true , address=145.79.227.19:4079 , piece_id=4CL7DQLWTXYVV7KQIFZWOH2AGULRFNXO2YN4MSPQAQ5P42GZH4DQ, seg_pos={0 0}, upload=507801]: Uploading piece... Dec 29 18:45:30.260767809 (579h16m30.131850855s) [noise=true , address=45.91.116.9:4261 , piece_id=E323JETYPKF377M6BY7ONX2F63Z7X5C3QWPXN2UPH4OXPAN4FNBA, seg_pos={0 0}, upload=507801]: Uploading piece... Dec 29 18:45:30.264585690 (579h16m30.135668725s) [noise=true , address=145.79.227.33:4871 , piece_id=ETYZPJ3MGSVKPKJHNTCPGZDO32RWCH7HTOKSRK3QIHUP6NLSH6LA, seg_pos={0 0}, upload=507801]: Uploading piece... Dec 29 18:45:30.265216781 (579h16m30.136299826s) [noise=true , address=45.91.116.7:4009 , piece_id=VLNL4NS4LXCQ7AQNP5GPBWSRKRFS7KRJ6U5SQ62KJKTPYFS2GZJQ, seg_pos={0 0}, upload=507801]: Uploading piece... Dec 29 18:45:30.268393119 (579h16m30.139476164s) [noise=true , address=45.91.116.5:4299 , piece_id=QZ6IZFTZTUJDJIT6W7G3JL7VAKRNLK35MLI6AXL5YQATHK7P5VYQ, seg_pos={0 0}, upload=507801]: Uploading piece... Dec 29 18:45:31.851451682 (579h16m31.722534727s) [upload=507801]: Next returned nil segment Dec 29 18:45:31.851468034 (579h16m31.722551079s) [upload=507801]: Waiting for error group managing segments... Dec 29 18:45:32.234275227 (579h16m32.105358262s) [noise=true , address=145.79.227.12:4765 , piece_id=EXHDKYB3KSQMBDXSX46EBQPBZUGY5FYJ22G2YTGO2W66DISTLD4A, seg_pos={0 0}, upload=507801]: Done uploading piece. err: <nil> Dec 29 18:45:32.249663528 (579h16m32.120746573s) [noise=true , address=145.79.227.11:4729 , piece_id=SHSSR2UTR4Z5QOPABQB2SYBA3JLQME6XCKKVEGTDHGQHY7R4Y62A, seg_pos={0 0}, upload=507801]: Done uploading piece. err: <nil> Dec 29 18:45:32.261536786 (579h16m32.132619832s) [noise=true , address=145.79.227.45:4229 , piece_id=HJNSE7MRTSLQWGSQCYLUCBSEDFEV3K64EB4DERKUXHLJYDR3UHIQ, seg_pos={0 0}, upload=507801]: Done uploading piece. err: <nil> Dec 29 18:45:32.261574740 (579h16m32.132657785s) [noise=true , address=145.79.227.5:4222 , piece_id=UC5LM3TSFWCW6JQOLYK5773UKU67CAN2AGEA4GMIPSPH6FI4VOXA, seg_pos={0 0}, upload=507801]: Done uploading piece. err: <nil> Dec 29 18:45:32.269788289 (579h16m32.140871324s) [noise=true , address=145.79.227.19:4079 , piece_id=4CL7DQLWTXYVV7KQIFZWOH2AGULRFNXO2YN4MSPQAQ5P42GZH4DQ, seg_pos={0 0}, upload=507801]: Done uploading piece. err: <nil> Dec 29 18:45:32.283356185 (579h16m32.15443923s) [noise=true , address=145.79.227.4:4186 , piece_id=VLYTHEBWPLMEAUF5LFJ4XXS4KMQDXHZK4TMAP4BETZF4DDGVFEXA, seg_pos={0 0}, upload=507801]: Done uploading piece. err: <nil> Dec 29 18:45:32.503571559 (579h16m32.374654594s) [noise=true , address=45.91.116.7:4009 , piece_id=VLNL4NS4LXCQ7AQNP5GPBWSRKRFS7KRJ6U5SQ62KJKTPYFS2GZJQ, seg_pos={0 0}, upload=507801]: Done uploading piece. err: <nil> Dec 29 18:45:32.504458778 (579h16m32.375541813s) [noise=true , address=45.91.116.12:4225 , piece_id=R2MWDEFNPMI63DXBICR3OINB7L3MYNPTXVNG7HOMILXOHPCJYMWQ, seg_pos={0 0}, upload=507801]: Done uploading piece. err: <nil> Dec 29 18:45:32.505792748 (579h16m32.376875793s) [noise=true , address=45.91.116.14:4192 , piece_id=SHBEQUDUNTS2VSBU7WSBNGDVV5SCENXGC3BLRIG32W4SCN5DEWSA, seg_pos={0 0}, upload=507801]: Done uploading piece. err: <nil> Dec 29 18:45:32.535311226 (579h16m32.406394271s) [noise=true , address=45.91.116.5:4299 , piece_id=QZ6IZFTZTUJDJIT6W7G3JL7VAKRNLK35MLI6AXL5YQATHK7P5VYQ, seg_pos={0 0}, upload=507801]: Done uploading piece. err: <nil> Dec 29 18:45:32.553243982 (579h16m32.424327027s) [noise=true , address=45.91.116.4:4939 , piece_id=BKAMYPCAXF24VHJRC5UAKPCA4XOUFBKMVOWFCFVD2VIJLPVKCKDQ, seg_pos={0 0}, upload=507801]: Done uploading piece. err: <nil> Dec 29 18:45:32.563846030 (579h16m32.434929075s) [noise=true , address=45.91.116.9:4261 , piece_id=E323JETYPKF377M6BY7ONX2F63Z7X5C3QWPXN2UPH4OXPAN4FNBA, seg_pos={0 0}, upload=507801]: Done uploading piece. err: <nil> Dec 29 18:45:32.583730774 (579h16m32.454813809s) [noise=true , address=77.90.31.12:4589 , piece_id=DPYMF7AEO7E6LDMI7MM27F7PAVWUWP3H3J55KRC7VGCWHEXULPTQ, seg_pos={0 0}, upload=507801]: Done uploading piece. err: <nil> Dec 29 18:45:32.583748938 (579h16m32.454831983s) [seg_pos={0 0}, upload=507801]: Segment reached optimal threshold of 13 pieces. Dec 29 18:45:32.584000086 (579h16m32.455083131s) [noise=true , address=145.79.227.49:4339 , piece_id=B3UJYKAMRVRWBDBDTU47GJBOC7VKZ377DQW363OKIJPT5Y6YUPJA, seg_pos={0 0}, upload=507801]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 29 18:45:32.584008423 (579h16m32.455091468s) [noise=true , address=145.79.227.49:4339 , piece_id=B3UJYKAMRVRWBDBDTU47GJBOC7VKZ377DQW363OKIJPT5Y6YUPJA, seg_pos={0 0}, upload=507801]: Long tail cancellation due to optimal threshold Dec 29 18:45:32.584113132 (579h16m32.455196207s) [noise=true , address=145.79.227.33:4871 , piece_id=ETYZPJ3MGSVKPKJHNTCPGZDO32RWCH7HTOKSRK3QIHUP6NLSH6LA, seg_pos={0 0}, upload=507801]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 29 18:45:32.584135263 (579h16m32.455218319s) [noise=true , address=145.79.227.33:4871 , piece_id=ETYZPJ3MGSVKPKJHNTCPGZDO32RWCH7HTOKSRK3QIHUP6NLSH6LA, seg_pos={0 0}, upload=507801]: Long tail cancellation due to optimal threshold Dec 29 18:45:32.584316669 (579h16m32.455399714s) [noise=true , address=45.91.116.6:4973 , piece_id=OR335JA6AUPEKUM6FV5OPYPCG5AH55DPCZDDS5VMTY6BRNM6H5EQ, seg_pos={0 0}, upload=507801]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 29 18:45:32.584324815 (579h16m32.45540785s) [noise=true , address=45.91.116.6:4973 , piece_id=OR335JA6AUPEKUM6FV5OPYPCG5AH55DPCZDDS5VMTY6BRNM6H5EQ, seg_pos={0 0}, upload=507801]: Long tail cancellation due to optimal threshold Dec 29 18:45:32.584352888 (579h16m32.455435923s) [seg_pos={0 0}, upload=507801]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:45:32.584430666 (579h16m32.455513712s) [upload=507801]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:45:32.584437499 (579h16m32.455520534s) [upload=507801]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:45:32.878593124 (579h16m32.749676159s) [upload=507801]: Done uploading segments... Dec 29 18:45:32.878612592 (579h16m32.749695627s) [upload=507801]: Upload finished. err: <nil> Duration: 30.004606562s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 27.316027635s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:45:30 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 27.316059725s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X.XXX..X.....X.X.....X...XX.X......XXX.XX.X.XX.......X..XXXX] duration: 30.003691658s trace: https://jaeger.dev.storj.io/trace/286224f30dcb00dd FAIL (2025-12-29 18:44:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: SNSBK46CGFGUQGGEYUTS6PZV6J7TRC77KS4BRAW5XOCMLXXRW74A) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: OHYGUQKOJBNEZFQUKFE7VPF5J6I3CMH4DDN52GU5ISDWQWYH4UJQ) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: DVEQODFVDXVUG6GS5NZWIVSRWG64RPLNZWXTDCJXDXYNKWYHVXIQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: VRBHPOLBYFB7LX7CHYNS56OP26N2WNWCCOI42MIFFLQUOT3L7ANA) context deadline exceeded ======LOGS====== Dec 29 18:44:29.085406105 (579h15m28.95648914s) [upload=507789]: Starting upload Dec 29 18:44:29.085459476 (579h15m28.956542511s) [upload=507789]: Done starting upload Dec 29 18:44:29.085557332 (579h15m28.956640367s) [upload=507789]: Uploading segments... Dec 29 18:44:29.085674365 (579h15m28.95675741s) [upload=507789]: Got next segment. Inline: false Dec 29 18:44:29.085735922 (579h15m28.956818967s) [upload=507789]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:44:29.085742865 (579h15m28.95682591s) [upload=507789]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:44:29.333736895 (579h15m29.20481996s) [seg_pos={0 0}, upload=507789]: Begin upload segment... Dec 29 18:44:29.333845321 (579h15m29.204928356s) [seg_pos={0 0}, upload=507789]: Done begin upload segment. Dec 29 18:44:29.333861521 (579h15m29.204944567s) [noise=true , address=45.91.116.10:4082 , piece_id=DVEQODFVDXVUG6GS5NZWIVSRWG64RPLNZWXTDCJXDXYNKWYHVXIQ, seg_pos={0 0}, upload=507789]: Uploading piece... Dec 29 18:44:29.333887080 (579h15m29.204970145s) [noise=true , address=145.79.227.46:4602 , piece_id=VOFJUCIQNB6EE2MQDV5GCJHR7V6SPOVQY2CTK4LSJTIP5BHM4ZIQ, seg_pos={0 0}, upload=507789]: Uploading piece... Dec 29 18:44:29.333955360 (579h15m29.205038406s) [noise=true , address=45.91.116.7:4009 , piece_id=NVP3OHPQTBMPUX2SFLNS3IGEAYIFLIEKF6RBSRCICNF5KR3QMBZQ, seg_pos={0 0}, upload=507789]: Uploading piece... Dec 29 18:44:29.337852151 (579h15m29.208935206s) [noise=true , address=45.91.116.5:4299 , piece_id=4EGY3JZ3NSTAIAD3HUZ2OBXFK7TDBR3RGA5BYFU7LLFL7F7OTZUA, seg_pos={0 0}, upload=507789]: Uploading piece... Dec 29 18:44:29.338233527 (579h15m29.209316562s) [noise=true , address=145.79.227.21:4475 , piece_id=M6O2OWQLZVVKW4MOZIP66UMX5ACSJJYWHJVQKJVI5JOXBJVYFJXA, seg_pos={0 0}, upload=507789]: Uploading piece... Dec 29 18:44:29.338698712 (579h15m29.209781757s) [noise=true , address=145.79.227.18:4007 , piece_id=7XKEGGNN5C4CDDRNJU3TZUJGTPIDPHTVFNU3P6ABD5AJBFOLLSHA, seg_pos={0 0}, upload=507789]: Uploading piece... Dec 29 18:44:29.339875042 (579h15m29.210958088s) [noise=true , address=45.91.116.6:4975 , piece_id=VRBHPOLBYFB7LX7CHYNS56OP26N2WNWCCOI42MIFFLQUOT3L7ANA, seg_pos={0 0}, upload=507789]: Uploading piece... Dec 29 18:44:29.340809302 (579h15m29.211892337s) [noise=true , address=145.79.227.51:4409 , piece_id=AYZM3KWGBSBYQV7U7UT5JXTK5NZBKZSUPPYN2KBECZP6O4R2VMQQ, seg_pos={0 0}, upload=507789]: Uploading piece... Dec 29 18:44:29.341400307 (579h15m29.212483352s) [noise=true , address=77.90.31.14:4663 , piece_id=SNSBK46CGFGUQGGEYUTS6PZV6J7TRC77KS4BRAW5XOCMLXXRW74A, seg_pos={0 0}, upload=507789]: Uploading piece... Dec 29 18:44:29.341995530 (579h15m29.213078565s) [noise=true , address=145.79.227.19:4079 , piece_id=KQSFPXVCHA2N24EVGUKTDCO5CDGNYDYQAKU5S65JR7YCJTM5UXRA, seg_pos={0 0}, upload=507789]: Uploading piece... Dec 29 18:44:29.343709073 (579h15m29.214792118s) [noise=true , address=77.90.31.10:4518 , piece_id=OHYGUQKOJBNEZFQUKFE7VPF5J6I3CMH4DDN52GU5ISDWQWYH4UJQ, seg_pos={0 0}, upload=507789]: Uploading piece... Dec 29 18:44:29.345128927 (579h15m29.216211962s) [noise=true , address=145.79.227.26:4653 , piece_id=G4B35CJC7Q6C7VRDLDOP636HPR4PSALCRPASOJXFTKKBOCZNOBLA, seg_pos={0 0}, upload=507789]: Uploading piece... Dec 29 18:44:29.346787876 (579h15m29.217870921s) [noise=true , address=45.91.116.9:4263 , piece_id=LZWVUSXBMTA5T5BP2MW4R2UUJJG5KEBPIDTHH6FEMINOFZXBDWJA, seg_pos={0 0}, upload=507789]: Uploading piece... Dec 29 18:44:29.348518511 (579h15m29.219601556s) [noise=true , address=145.79.227.50:4375 , piece_id=OMFAS6MYVSGFLW6QLSBP2L43LZJUWUI4UBMXRPAQGQNKR2PGVQIA, seg_pos={0 0}, upload=507789]: Uploading piece... Dec 29 18:44:29.350156931 (579h15m29.221239966s) [noise=true , address=45.91.116.14:4191 , piece_id=CSYC4Q6ZUCRLGFSVDNKI6UVDYTKSI4XHRQ67LC4WWBMXCHTKSYZQ, seg_pos={0 0}, upload=507789]: Uploading piece... Dec 29 18:44:29.351471785 (579h15m29.22255482s) [noise=true , address=145.79.227.29:4942 , piece_id=4DRK2CQPFGLODGG7JNBJH7ZK7NEO233IEHCJBAHZOXDCDCZXY3GQ, seg_pos={0 0}, upload=507789]: Uploading piece... Dec 29 18:44:31.034219503 (579h15m30.905302548s) [upload=507789]: Next returned nil segment Dec 29 18:44:31.034240162 (579h15m30.905323197s) [upload=507789]: Waiting for error group managing segments... Dec 29 18:44:31.314165833 (579h15m31.185248868s) [noise=true , address=145.79.227.19:4079 , piece_id=KQSFPXVCHA2N24EVGUKTDCO5CDGNYDYQAKU5S65JR7YCJTM5UXRA, seg_pos={0 0}, upload=507789]: Done uploading piece. err: <nil> Dec 29 18:44:31.320971141 (579h15m31.192054176s) [noise=true , address=145.79.227.18:4007 , piece_id=7XKEGGNN5C4CDDRNJU3TZUJGTPIDPHTVFNU3P6ABD5AJBFOLLSHA, seg_pos={0 0}, upload=507789]: Done uploading piece. err: <nil> Dec 29 18:44:31.321663479 (579h15m31.192746515s) [noise=true , address=145.79.227.21:4475 , piece_id=M6O2OWQLZVVKW4MOZIP66UMX5ACSJJYWHJVQKJVI5JOXBJVYFJXA, seg_pos={0 0}, upload=507789]: Done uploading piece. err: <nil> Dec 29 18:44:31.335497972 (579h15m31.206581017s) [noise=true , address=145.79.227.46:4602 , piece_id=VOFJUCIQNB6EE2MQDV5GCJHR7V6SPOVQY2CTK4LSJTIP5BHM4ZIQ, seg_pos={0 0}, upload=507789]: Done uploading piece. err: <nil> Dec 29 18:44:31.348980736 (579h15m31.220063771s) [noise=true , address=145.79.227.26:4653 , piece_id=G4B35CJC7Q6C7VRDLDOP636HPR4PSALCRPASOJXFTKKBOCZNOBLA, seg_pos={0 0}, upload=507789]: Done uploading piece. err: <nil> Dec 29 18:44:31.359114181 (579h15m31.230197216s) [noise=true , address=145.79.227.51:4409 , piece_id=AYZM3KWGBSBYQV7U7UT5JXTK5NZBKZSUPPYN2KBECZP6O4R2VMQQ, seg_pos={0 0}, upload=507789]: Done uploading piece. err: <nil> Dec 29 18:44:31.582331441 (579h15m31.453414486s) [noise=true , address=77.90.31.10:4518 , piece_id=OHYGUQKOJBNEZFQUKFE7VPF5J6I3CMH4DDN52GU5ISDWQWYH4UJQ, seg_pos={0 0}, upload=507789]: Done uploading piece. err: <nil> Dec 29 18:44:31.586724296 (579h15m31.457807341s) [noise=true , address=45.91.116.7:4009 , piece_id=NVP3OHPQTBMPUX2SFLNS3IGEAYIFLIEKF6RBSRCICNF5KR3QMBZQ, seg_pos={0 0}, upload=507789]: Done uploading piece. err: <nil> Dec 29 18:44:31.626614112 (579h15m31.497697167s) [noise=true , address=45.91.116.6:4975 , piece_id=VRBHPOLBYFB7LX7CHYNS56OP26N2WNWCCOI42MIFFLQUOT3L7ANA, seg_pos={0 0}, upload=507789]: Done uploading piece. err: <nil> Dec 29 18:44:31.705343164 (579h15m31.576426209s) [noise=true , address=45.91.116.14:4191 , piece_id=CSYC4Q6ZUCRLGFSVDNKI6UVDYTKSI4XHRQ67LC4WWBMXCHTKSYZQ, seg_pos={0 0}, upload=507789]: Done uploading piece. err: <nil> Dec 29 18:44:31.729639278 (579h15m31.600722323s) [noise=true , address=45.91.116.9:4263 , piece_id=LZWVUSXBMTA5T5BP2MW4R2UUJJG5KEBPIDTHH6FEMINOFZXBDWJA, seg_pos={0 0}, upload=507789]: Done uploading piece. err: <nil> Dec 29 18:44:32.462448999 (579h15m32.333532034s) [noise=true , address=45.91.116.10:4082 , piece_id=DVEQODFVDXVUG6GS5NZWIVSRWG64RPLNZWXTDCJXDXYNKWYHVXIQ, seg_pos={0 0}, upload=507789]: Done uploading piece. err: <nil> Dec 29 18:44:32.510668807 (579h15m32.381751842s) [noise=true , address=77.90.31.14:4663 , piece_id=SNSBK46CGFGUQGGEYUTS6PZV6J7TRC77KS4BRAW5XOCMLXXRW74A, seg_pos={0 0}, upload=507789]: Done uploading piece. err: <nil> Dec 29 18:44:32.510691250 (579h15m32.381774285s) [seg_pos={0 0}, upload=507789]: Segment reached optimal threshold of 13 pieces. Dec 29 18:44:32.510881331 (579h15m32.381964366s) [noise=true , address=145.79.227.50:4375 , piece_id=OMFAS6MYVSGFLW6QLSBP2L43LZJUWUI4UBMXRPAQGQNKR2PGVQIA, seg_pos={0 0}, upload=507789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF Dec 29 18:44:32.510888095 (579h15m32.38197113s) [noise=true , address=145.79.227.50:4375 , piece_id=OMFAS6MYVSGFLW6QLSBP2L43LZJUWUI4UBMXRPAQGQNKR2PGVQIA, seg_pos={0 0}, upload=507789]: Long tail cancellation due to optimal threshold Dec 29 18:44:32.511071704 (579h15m32.382154739s) [noise=true , address=45.91.116.5:4299 , piece_id=4EGY3JZ3NSTAIAD3HUZ2OBXFK7TDBR3RGA5BYFU7LLFL7F7OTZUA, seg_pos={0 0}, upload=507789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 29 18:44:32.511077565 (579h15m32.3821606s) [noise=true , address=45.91.116.5:4299 , piece_id=4EGY3JZ3NSTAIAD3HUZ2OBXFK7TDBR3RGA5BYFU7LLFL7F7OTZUA, seg_pos={0 0}, upload=507789]: Long tail cancellation due to optimal threshold Dec 29 18:44:32.511173689 (579h15m32.382256724s) [noise=true , address=145.79.227.29:4942 , piece_id=4DRK2CQPFGLODGG7JNBJH7ZK7NEO233IEHCJBAHZOXDCDCZXY3GQ, seg_pos={0 0}, upload=507789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 29 18:44:32.511190941 (579h15m32.382273976s) [noise=true , address=145.79.227.29:4942 , piece_id=4DRK2CQPFGLODGG7JNBJH7ZK7NEO233IEHCJBAHZOXDCDCZXY3GQ, seg_pos={0 0}, upload=507789]: Long tail cancellation due to optimal threshold Dec 29 18:44:32.511217963 (579h15m32.382300998s) [seg_pos={0 0}, upload=507789]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:44:32.511349684 (579h15m32.382432719s) [upload=507789]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:44:32.511355294 (579h15m32.382438329s) [upload=507789]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:44:32.831330165 (579h15m32.70241321s) [upload=507789]: Done uploading segments... Dec 29 18:44:32.831349401 (579h15m32.702432436s) [upload=507789]: Upload finished. err: <nil> Duration: 30.003704152s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.X.XXX..X.....X.X.....X...XX.X......XXX.XX.X.XX.......X..XXX] duration: 30.003015409s trace: https://jaeger.dev.storj.io/trace/347a0e66a4adc131 FAIL (2025-12-29 18:44:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 18:43:58.845608877 (579h14m58.716691912s) [upload=507782]: Starting upload Dec 29 18:43:58.845663341 (579h14m58.716746376s) [upload=507782]: Done starting upload Dec 29 18:43:58.845755716 (579h14m58.716838761s) [upload=507782]: Uploading segments... Dec 29 18:43:58.845904560 (579h14m58.716987595s) [upload=507782]: Got next segment. Inline: false Dec 29 18:43:58.845937944 (579h14m58.717020979s) [upload=507782]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:43:58.845943474 (579h14m58.717026509s) [upload=507782]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:43:59.102958720 (579h14m58.974041755s) [seg_pos={0 0}, upload=507782]: Begin upload segment... Dec 29 18:43:59.103014576 (579h14m58.974097611s) [seg_pos={0 0}, upload=507782]: Done begin upload segment. Dec 29 18:43:59.103052028 (579h14m58.974135063s) [noise=true , address=45.91.116.14:4192 , piece_id=3HWMHIGS7EFFXMSVPSQ2METAEORLEAEBS7JGX5A46DMCCCWE5OIQ, seg_pos={0 0}, upload=507782]: Uploading piece... Dec 29 18:43:59.103161937 (579h14m58.974244982s) [noise=true , address=145.79.227.35:4482 , piece_id=DZ3KKNC3W7OSMFNAGAONKNILWJFH35C7622RFLP23KZDLXCRFBRA, seg_pos={0 0}, upload=507782]: Uploading piece... Dec 29 18:43:59.103864504 (579h14m58.974947549s) [noise=true , address=145.79.227.44:4194 , piece_id=5WX6VRGQNUWG3CYOH46SDEHRMAV6D5SUC6RHCRGAKNUHGBVPRFYA, seg_pos={0 0}, upload=507782]: Uploading piece... Dec 29 18:43:59.104001415 (579h14m58.97508445s) [noise=true , address=77.90.31.14:4661 , piece_id=FBMFPP3PPBCNSU2IU3I5LOYWFX4EASTPC2VFCXXPSSFIRZP3YISA, seg_pos={0 0}, upload=507782]: Uploading piece... Dec 29 18:43:59.104491899 (579h14m58.975574934s) [noise=true , address=145.79.227.17:4972 , piece_id=SQ7D3Y6FOA53Z5YRMTBCFIVQZHN6XSNUCWJHLDYIABVQ36KU4CIQ, seg_pos={0 0}, upload=507782]: Uploading piece... Dec 29 18:43:59.105347008 (579h14m58.976430043s) [noise=true , address=45.91.116.9:4264 , piece_id=MLD4YWZ53STUN3QW4SSWAA5HPVFX7PFFVC45TL4QFNH4GXBPMADQ, seg_pos={0 0}, upload=507782]: Uploading piece... Dec 29 18:43:59.105395700 (579h14m58.976478745s) [noise=true , address=77.90.31.11:4555 , piece_id=WLVJH6I7RY4JRSMFIPD6VFE5I37MIR6EH4MSLW5QRBNDYXM6OC5A, seg_pos={0 0}, upload=507782]: Uploading piece... Dec 29 18:43:59.105987827 (579h14m58.977070873s) [noise=true , address=145.79.227.7:4295 , piece_id=KRYEJUVZF3TCVZERFOIEHPKKEHP36GALEICAWKHI32RS4P5PXCSA, seg_pos={0 0}, upload=507782]: Uploading piece... Dec 29 18:43:59.106422646 (579h14m58.977505681s) [noise=true , address=145.79.227.15:4873 , piece_id=X7JNIPBB77LGHIAPSNIS7DB4HMP3SYWDEYMEHPSBNU7D54C2EXOQ, seg_pos={0 0}, upload=507782]: Uploading piece... Dec 29 18:43:59.106821505 (579h14m58.97790455s) [noise=true , address=45.91.116.10:4083 , piece_id=HJQOJVJTHV2H46Q5LOY2OJWZHWYFIOM4C5CBPGXTUVHGJZTOT6SQ, seg_pos={0 0}, upload=507782]: Uploading piece... Dec 29 18:43:59.107190498 (579h14m58.978273543s) [noise=true , address=145.79.227.51:4411 , piece_id=NGOBFKINEBX4DDFB2IAWKSDTCGAFXSULOFMTC2ZQQ6HVZQM3CS5A, seg_pos={0 0}, upload=507782]: Uploading piece... Dec 29 18:43:59.107782345 (579h14m58.97886538s) [noise=true , address=145.79.227.4:4185 , piece_id=K2J4GZXJOEPHAIXDE2BJT66XQAMLQCHT7FJGZU54HWYEX3HF5LSA, seg_pos={0 0}, upload=507782]: Uploading piece... Dec 29 18:43:59.108296905 (579h14m58.97937994s) [noise=true , address=145.79.227.26:4654 , piece_id=H6JU7XLOHRRYRDM4QKL4YI4KJVB3PT7J4QWQEWVT3UODZY6LAD4A, seg_pos={0 0}, upload=507782]: Uploading piece... Dec 29 18:43:59.109437076 (579h14m58.980520111s) [noise=true , address=145.79.227.19:4077 , piece_id=C4C4QVSS4B23QLIY7J2PEZQGADDBAVC26WV4XEYBCGNFZLNCKWTQ, seg_pos={0 0}, upload=507782]: Uploading piece... Dec 29 18:43:59.111005072 (579h14m58.982088107s) [noise=true , address=77.90.31.18:4805 , piece_id=7YC63LE3GOLSNZRMARW42GGNIEGMIY6YWEFZ4UZNUONWV2LDXPCQ, seg_pos={0 0}, upload=507782]: Uploading piece... Dec 29 18:43:59.112011177 (579h14m58.983094212s) [noise=true , address=45.91.116.6:4975 , piece_id=HOOOKN7U2TS2F6EIWOKE3ZODEPAQFSFCBFJL6CPBC6WJRBB5JFCA, seg_pos={0 0}, upload=507782]: Uploading piece... Dec 29 18:44:00.806827116 (579h15m0.677910151s) [upload=507782]: Next returned nil segment Dec 29 18:44:00.806845661 (579h15m0.677928696s) [upload=507782]: Waiting for error group managing segments... Dec 29 18:44:01.088383633 (579h15m0.959466678s) [noise=true , address=145.79.227.4:4185 , piece_id=K2J4GZXJOEPHAIXDE2BJT66XQAMLQCHT7FJGZU54HWYEX3HF5LSA, seg_pos={0 0}, upload=507782]: Done uploading piece. err: <nil> Dec 29 18:44:01.093613441 (579h15m0.964696486s) [noise=true , address=145.79.227.17:4972 , piece_id=SQ7D3Y6FOA53Z5YRMTBCFIVQZHN6XSNUCWJHLDYIABVQ36KU4CIQ, seg_pos={0 0}, upload=507782]: Done uploading piece. err: <nil> Dec 29 18:44:01.106339082 (579h15m0.977422127s) [noise=true , address=145.79.227.7:4295 , piece_id=KRYEJUVZF3TCVZERFOIEHPKKEHP36GALEICAWKHI32RS4P5PXCSA, seg_pos={0 0}, upload=507782]: Done uploading piece. err: <nil> Dec 29 18:44:01.108290398 (579h15m0.979373443s) [noise=true , address=145.79.227.15:4873 , piece_id=X7JNIPBB77LGHIAPSNIS7DB4HMP3SYWDEYMEHPSBNU7D54C2EXOQ, seg_pos={0 0}, upload=507782]: Done uploading piece. err: <nil> Dec 29 18:44:01.112039106 (579h15m0.983122151s) [noise=true , address=145.79.227.19:4077 , piece_id=C4C4QVSS4B23QLIY7J2PEZQGADDBAVC26WV4XEYBCGNFZLNCKWTQ, seg_pos={0 0}, upload=507782]: Done uploading piece. err: <nil> Dec 29 18:44:01.124790687 (579h15m0.995873732s) [noise=true , address=145.79.227.51:4411 , piece_id=NGOBFKINEBX4DDFB2IAWKSDTCGAFXSULOFMTC2ZQQ6HVZQM3CS5A, seg_pos={0 0}, upload=507782]: Done uploading piece. err: <nil> Dec 29 18:44:01.128784301 (579h15m0.999867336s) [noise=true , address=145.79.227.26:4654 , piece_id=H6JU7XLOHRRYRDM4QKL4YI4KJVB3PT7J4QWQEWVT3UODZY6LAD4A, seg_pos={0 0}, upload=507782]: Done uploading piece. err: <nil> Dec 29 18:44:01.318568890 (579h15m1.189651935s) [noise=true , address=77.90.31.11:4555 , piece_id=WLVJH6I7RY4JRSMFIPD6VFE5I37MIR6EH4MSLW5QRBNDYXM6OC5A, seg_pos={0 0}, upload=507782]: Done uploading piece. err: <nil> Dec 29 18:44:01.325917432 (579h15m1.197000477s) [noise=true , address=45.91.116.14:4192 , piece_id=3HWMHIGS7EFFXMSVPSQ2METAEORLEAEBS7JGX5A46DMCCCWE5OIQ, seg_pos={0 0}, upload=507782]: Done uploading piece. err: <nil> Dec 29 18:44:01.337158277 (579h15m1.208241322s) [noise=true , address=77.90.31.14:4661 , piece_id=FBMFPP3PPBCNSU2IU3I5LOYWFX4EASTPC2VFCXXPSSFIRZP3YISA, seg_pos={0 0}, upload=507782]: Done uploading piece. err: <nil> Dec 29 18:44:01.476512488 (579h15m1.347595533s) [noise=true , address=45.91.116.10:4083 , piece_id=HJQOJVJTHV2H46Q5LOY2OJWZHWYFIOM4C5CBPGXTUVHGJZTOT6SQ, seg_pos={0 0}, upload=507782]: Done uploading piece. err: <nil> Dec 29 18:44:01.965399543 (579h15m1.836482589s) [noise=true , address=77.90.31.18:4805 , piece_id=7YC63LE3GOLSNZRMARW42GGNIEGMIY6YWEFZ4UZNUONWV2LDXPCQ, seg_pos={0 0}, upload=507782]: Done uploading piece. err: <nil> Dec 29 18:44:02.156038740 (579h15m2.027121775s) [noise=true , address=45.91.116.6:4975 , piece_id=HOOOKN7U2TS2F6EIWOKE3ZODEPAQFSFCBFJL6CPBC6WJRBB5JFCA, seg_pos={0 0}, upload=507782]: Done uploading piece. err: <nil> Dec 29 18:44:02.156064630 (579h15m2.027147675s) [seg_pos={0 0}, upload=507782]: Segment reached optimal threshold of 13 pieces. Dec 29 18:44:02.156324173 (579h15m2.027407248s) [noise=true , address=45.91.116.9:4264 , piece_id=MLD4YWZ53STUN3QW4SSWAA5HPVFX7PFFVC45TL4QFNH4GXBPMADQ, seg_pos={0 0}, upload=507782]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:36818->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:36818->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36818->45.91.116.9:4264: use of closed network connection Dec 29 18:44:02.156334693 (579h15m2.027417738s) [noise=true , address=45.91.116.9:4264 , piece_id=MLD4YWZ53STUN3QW4SSWAA5HPVFX7PFFVC45TL4QFNH4GXBPMADQ, seg_pos={0 0}, upload=507782]: Long tail cancellation due to optimal threshold Dec 29 18:44:02.156573358 (579h15m2.027656403s) [noise=true , address=145.79.227.44:4194 , piece_id=5WX6VRGQNUWG3CYOH46SDEHRMAV6D5SUC6RHCRGAKNUHGBVPRFYA, seg_pos={0 0}, upload=507782]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 18:44:02.156581413 (579h15m2.027664458s) [noise=true , address=145.79.227.44:4194 , piece_id=5WX6VRGQNUWG3CYOH46SDEHRMAV6D5SUC6RHCRGAKNUHGBVPRFYA, seg_pos={0 0}, upload=507782]: Long tail cancellation due to optimal threshold Dec 29 18:44:02.156778218 (579h15m2.027861273s) [noise=true , address=145.79.227.35:4482 , piece_id=DZ3KKNC3W7OSMFNAGAONKNILWJFH35C7622RFLP23KZDLXCRFBRA, seg_pos={0 0}, upload=507782]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 29 18:44:02.156807884 (579h15m2.02789093s) [noise=true , address=145.79.227.35:4482 , piece_id=DZ3KKNC3W7OSMFNAGAONKNILWJFH35C7622RFLP23KZDLXCRFBRA, seg_pos={0 0}, upload=507782]: Long tail cancellation due to optimal threshold Dec 29 18:44:02.156867558 (579h15m2.027950603s) [seg_pos={0 0}, upload=507782]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:44:02.157060986 (579h15m2.028144051s) [upload=507782]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:44:02.157087848 (579h15m2.028170893s) [upload=507782]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:44:02.470842625 (579h15m2.34192568s) [upload=507782]: Done uploading segments... Dec 29 18:44:02.470871690 (579h15m2.341954735s) [upload=507782]: Upload finished. err: <nil> Duration: 30.00303733s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 24.356871532s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:44:26 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 24.356887992s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 3.430672361s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:44:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.430696286s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 35.602720844s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:44:01 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 35.602755368s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 3.469412096s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:43:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.469443617s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..X.XXX..X.....X.X.....X...XX.X......XXX.XX.X.XX.......X..XX] duration: 30.002897192s trace: https://jaeger.dev.storj.io/trace/2f6f2fcdcbdad791 FAIL (2025-12-29 18:43:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 18:42:56.928321377 (579h13m56.799404422s) [upload=507769]: Starting upload Dec 29 18:42:56.928397422 (579h13m56.799480467s) [upload=507769]: Done starting upload Dec 29 18:42:56.928439762 (579h13m56.799522797s) [upload=507769]: Uploading segments... Dec 29 18:42:56.928578917 (579h13m56.799661962s) [upload=507769]: Got next segment. Inline: false Dec 29 18:42:56.928604967 (579h13m56.799688012s) [upload=507769]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:42:56.928609205 (579h13m56.79969224s) [upload=507769]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:42:57.187127133 (579h13m57.058210178s) [seg_pos={0 0}, upload=507769]: Begin upload segment... Dec 29 18:42:57.187181006 (579h13m57.058264041s) [seg_pos={0 0}, upload=507769]: Done begin upload segment. Dec 29 18:42:57.187224909 (579h13m57.058307954s) [noise=true , address=145.79.227.21:4475 , piece_id=QF7CIKFCLWYQSU6K5LA4Y5UTIVWIVI3FDCVDNBKEA5YSF5V4VLKA, seg_pos={0 0}, upload=507769]: Uploading piece... Dec 29 18:42:57.187246300 (579h13m57.058329335s) [noise=true , address=145.79.227.41:4087 , piece_id=L4FLE732UOZJF35YUM6HB7LJG6ZATWCPN4XHL6ZYH6NK63VNR7WQ, seg_pos={0 0}, upload=507769]: Uploading piece... Dec 29 18:42:57.187654306 (579h13m57.058737351s) [noise=true , address=45.91.116.13:4156 , piece_id=DD2XLAIUOU4DOEEGMKX4YQWOITGJ5UDEQBCCLUCSIANAILGSWQPA, seg_pos={0 0}, upload=507769]: Uploading piece... Dec 29 18:42:57.188005525 (579h13m57.05908856s) [noise=true , address=145.79.227.11:4730 , piece_id=EKTFZB4SBH7D5BLXXZZVSZ4AOVMN3TBYBBUYH4RR3CYACRODBB7Q, seg_pos={0 0}, upload=507769]: Uploading piece... Dec 29 18:42:57.188051674 (579h13m57.059134709s) [noise=true , address=77.90.31.11:4553 , piece_id=U65ZF7JRESD55A4WO4HPXTYUFEKZLEZHWYQDWTOLDFU63FGVU2DA, seg_pos={0 0}, upload=507769]: Uploading piece... Dec 29 18:42:57.188365711 (579h13m57.059448747s) [noise=true , address=145.79.227.10:4402 , piece_id=7XWK3NUMT27776NGXDFFUU3CECOJJGIGFWNTKEM3KKLTW7SUBF3Q, seg_pos={0 0}, upload=507769]: Uploading piece... Dec 29 18:42:57.188409785 (579h13m57.05949282s) [noise=true , address=145.79.227.45:4229 , piece_id=IVPNQTWATZNVWRK7QUE5HCW4RBVSERPYAPVCXYYYVJDTH3EJ4VDQ, seg_pos={0 0}, upload=507769]: Uploading piece... Dec 29 18:42:57.188674389 (579h13m57.059757424s) [noise=true , address=145.79.227.3:4149 , piece_id=NLDVREIRGGPTAU6ALBTQWFATJSXQ7TGGY4G7BXCJQEJ2JQZ2MNYQ, seg_pos={0 0}, upload=507769]: Uploading piece... Dec 29 18:42:57.188710759 (579h13m57.059793794s) [noise=true , address=77.90.31.18:4805 , piece_id=477C47D3LV3BBAMFLZX6A2YERH3EXEUBDWYAUE7OIUAMYLHPFQVQ, seg_pos={0 0}, upload=507769]: Uploading piece... Dec 29 18:42:57.188922882 (579h13m57.060005928s) [noise=true , address=77.90.31.14:4662 , piece_id=JRRFYFW2XV7YZI6COZJM7E7R5HREBHMFNEG2ZPNHGBABW23QTFBQ, seg_pos={0 0}, upload=507769]: Uploading piece... Dec 29 18:42:57.189072397 (579h13m57.060155432s) [noise=true , address=145.79.227.6:4257 , piece_id=336JHKXJGANFDK2QLJJGVWBKOCEXYCMGEAEHSFDC7D7GRV5UIHWQ, seg_pos={0 0}, upload=507769]: Uploading piece... Dec 29 18:42:57.189434576 (579h13m57.060517621s) [noise=true , address=145.79.227.14:4838 , piece_id=DCZKUNRL4PKXT3FKQRIAGMDSNQ2LDXFM7LVJEL2RLCJNG3WZV6BQ, seg_pos={0 0}, upload=507769]: Uploading piece... Dec 29 18:42:57.189874715 (579h13m57.06095776s) [noise=true , address=145.79.227.15:4874 , piece_id=OKGSR57IZ6Z5OA4W3FRHBZEW3NHJWH3C26MLP5GUR46VY7XTGQDA, seg_pos={0 0}, upload=507769]: Uploading piece... Dec 29 18:42:57.190407669 (579h13m57.061490704s) [noise=true , address=145.79.227.29:4942 , piece_id=OG3EUZLYPYBE2EX5QQEEJUMPYOUYARS7XTD726HKHYNDH7ULMEGQ, seg_pos={0 0}, upload=507769]: Uploading piece... Dec 29 18:42:57.190834522 (579h13m57.061917557s) [noise=true , address=145.79.227.13:4804 , piece_id=ILSYD6BCJEN5AJSAR6JSDDNN6LBOAJNOS4RCUHJST5ZRTX52CXZA, seg_pos={0 0}, upload=507769]: Uploading piece... Dec 29 18:42:57.191017761 (579h13m57.062100796s) [noise=true , address=145.79.227.46:4601 , piece_id=A7RZLPUTIYSLGF2SQA6A6Z2IUEVQFI277IOEKFQL4FKGNFJX2A5A, seg_pos={0 0}, upload=507769]: Uploading piece... Dec 29 18:42:58.776334415 (579h13m58.64741745s) [upload=507769]: Next returned nil segment Dec 29 18:42:58.776347439 (579h13m58.647430474s) [upload=507769]: Waiting for error group managing segments... Dec 29 18:42:59.157314498 (579h13m59.028397533s) [noise=true , address=145.79.227.15:4874 , piece_id=OKGSR57IZ6Z5OA4W3FRHBZEW3NHJWH3C26MLP5GUR46VY7XTGQDA, seg_pos={0 0}, upload=507769]: Done uploading piece. err: <nil> Dec 29 18:42:59.168724575 (579h13m59.03980761s) [noise=true , address=145.79.227.6:4257 , piece_id=336JHKXJGANFDK2QLJJGVWBKOCEXYCMGEAEHSFDC7D7GRV5UIHWQ, seg_pos={0 0}, upload=507769]: Done uploading piece. err: <nil> Dec 29 18:42:59.172022626 (579h13m59.043105661s) [noise=true , address=145.79.227.45:4229 , piece_id=IVPNQTWATZNVWRK7QUE5HCW4RBVSERPYAPVCXYYYVJDTH3EJ4VDQ, seg_pos={0 0}, upload=507769]: Done uploading piece. err: <nil> Dec 29 18:42:59.176430539 (579h13m59.047513574s) [noise=true , address=145.79.227.3:4149 , piece_id=NLDVREIRGGPTAU6ALBTQWFATJSXQ7TGGY4G7BXCJQEJ2JQZ2MNYQ, seg_pos={0 0}, upload=507769]: Done uploading piece. err: <nil> Dec 29 18:42:59.183445145 (579h13m59.05452818s) [noise=true , address=145.79.227.10:4402 , piece_id=7XWK3NUMT27776NGXDFFUU3CECOJJGIGFWNTKEM3KKLTW7SUBF3Q, seg_pos={0 0}, upload=507769]: Done uploading piece. err: <nil> Dec 29 18:42:59.187142356 (579h13m59.058225401s) [noise=true , address=145.79.227.11:4730 , piece_id=EKTFZB4SBH7D5BLXXZZVSZ4AOVMN3TBYBBUYH4RR3CYACRODBB7Q, seg_pos={0 0}, upload=507769]: Done uploading piece. err: <nil> Dec 29 18:42:59.193214358 (579h13m59.064297393s) [noise=true , address=145.79.227.41:4087 , piece_id=L4FLE732UOZJF35YUM6HB7LJG6ZATWCPN4XHL6ZYH6NK63VNR7WQ, seg_pos={0 0}, upload=507769]: Done uploading piece. err: <nil> Dec 29 18:42:59.196883624 (579h13m59.067966659s) [noise=true , address=145.79.227.13:4804 , piece_id=ILSYD6BCJEN5AJSAR6JSDDNN6LBOAJNOS4RCUHJST5ZRTX52CXZA, seg_pos={0 0}, upload=507769]: Done uploading piece. err: <nil> Dec 29 18:42:59.198363872 (579h13m59.069446907s) [noise=true , address=145.79.227.46:4601 , piece_id=A7RZLPUTIYSLGF2SQA6A6Z2IUEVQFI277IOEKFQL4FKGNFJX2A5A, seg_pos={0 0}, upload=507769]: Done uploading piece. err: <nil> Dec 29 18:42:59.199970072 (579h13m59.071053107s) [noise=true , address=145.79.227.21:4475 , piece_id=QF7CIKFCLWYQSU6K5LA4Y5UTIVWIVI3FDCVDNBKEA5YSF5V4VLKA, seg_pos={0 0}, upload=507769]: Done uploading piece. err: <nil> Dec 29 18:42:59.209295509 (579h13m59.080378564s) [noise=true , address=145.79.227.14:4838 , piece_id=DCZKUNRL4PKXT3FKQRIAGMDSNQ2LDXFM7LVJEL2RLCJNG3WZV6BQ, seg_pos={0 0}, upload=507769]: Done uploading piece. err: <nil> Dec 29 18:42:59.337017641 (579h13m59.208100676s) [noise=true , address=77.90.31.11:4553 , piece_id=U65ZF7JRESD55A4WO4HPXTYUFEKZLEZHWYQDWTOLDFU63FGVU2DA, seg_pos={0 0}, upload=507769]: Done uploading piece. err: <nil> Dec 29 18:42:59.342283658 (579h13m59.213366693s) [noise=true , address=77.90.31.14:4662 , piece_id=JRRFYFW2XV7YZI6COZJM7E7R5HREBHMFNEG2ZPNHGBABW23QTFBQ, seg_pos={0 0}, upload=507769]: Done uploading piece. err: <nil> Dec 29 18:42:59.342302364 (579h13m59.213385399s) [seg_pos={0 0}, upload=507769]: Segment reached optimal threshold of 13 pieces. Dec 29 18:42:59.342492215 (579h13m59.21357525s) [noise=true , address=45.91.116.13:4156 , piece_id=DD2XLAIUOU4DOEEGMKX4YQWOITGJ5UDEQBCCLUCSIANAILGSWQPA, seg_pos={0 0}, upload=507769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 29 18:42:59.342499268 (579h13m59.213582304s) [noise=true , address=45.91.116.13:4156 , piece_id=DD2XLAIUOU4DOEEGMKX4YQWOITGJ5UDEQBCCLUCSIANAILGSWQPA, seg_pos={0 0}, upload=507769]: Long tail cancellation due to optimal threshold Dec 29 18:42:59.342669533 (579h13m59.213752568s) [noise=true , address=145.79.227.29:4942 , piece_id=OG3EUZLYPYBE2EX5QQEEJUMPYOUYARS7XTD726HKHYNDH7ULMEGQ, seg_pos={0 0}, upload=507769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Dec 29 18:42:59.342684923 (579h13m59.213767958s) [noise=true , address=145.79.227.29:4942 , piece_id=OG3EUZLYPYBE2EX5QQEEJUMPYOUYARS7XTD726HKHYNDH7ULMEGQ, seg_pos={0 0}, upload=507769]: Long tail cancellation due to optimal threshold Dec 29 18:42:59.342777298 (579h13m59.213860333s) [noise=true , address=77.90.31.18:4805 , piece_id=477C47D3LV3BBAMFLZX6A2YERH3EXEUBDWYAUE7OIUAMYLHPFQVQ, seg_pos={0 0}, upload=507769]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 29 18:42:59.342785484 (579h13m59.213868519s) [noise=true , address=77.90.31.18:4805 , piece_id=477C47D3LV3BBAMFLZX6A2YERH3EXEUBDWYAUE7OIUAMYLHPFQVQ, seg_pos={0 0}, upload=507769]: Long tail cancellation due to optimal threshold Dec 29 18:42:59.342803498 (579h13m59.213886533s) [seg_pos={0 0}, upload=507769]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:42:59.342920962 (579h13m59.214003997s) [upload=507769]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:42:59.342927485 (579h13m59.21401052s) [upload=507769]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:42:59.656269485 (579h13m59.52735252s) [upload=507769]: Done uploading segments... Dec 29 18:42:59.656290826 (579h13m59.527373861s) [upload=507769]: Upload finished. err: <nil> Duration: 30.002919124s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 3.421258303s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:43:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.421285525s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 30.032753061s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:42:53 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 30.032769712s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 31.333195415s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:42:23 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 31.333220322s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X..X.XXX..X.....X.X.....X...XX.X......XXX.XX.X.XX.......X..X] duration: 30.003749728s trace: https://jaeger.dev.storj.io/trace/5d99f245ca2fb9d8 FAIL (2025-12-29 18:42:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 29 18:41:50.216141708 (579h12m50.087224743s) [upload=507753]: Starting upload Dec 29 18:41:50.216205891 (579h12m50.087288926s) [upload=507753]: Done starting upload Dec 29 18:41:50.216234145 (579h12m50.08731718s) [upload=507753]: Uploading segments... Dec 29 18:41:50.216489611 (579h12m50.087572666s) [upload=507753]: Got next segment. Inline: false Dec 29 18:41:50.216532001 (579h12m50.087615036s) [upload=507753]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:41:50.216537021 (579h12m50.087620066s) [upload=507753]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:41:50.463270709 (579h12m50.334353744s) [seg_pos={0 0}, upload=507753]: Begin upload segment... Dec 29 18:41:50.463331064 (579h12m50.33441411s) [seg_pos={0 0}, upload=507753]: Done begin upload segment. Dec 29 18:41:50.463379517 (579h12m50.334462572s) [noise=true , address=77.90.31.18:4808 , piece_id=O2APWAJ7OUPHWV6W2MOETRITDK7SA5CT5HLCZNCQRXVDXN6TEM4A, seg_pos={0 0}, upload=507753]: Uploading piece... Dec 29 18:41:50.463393704 (579h12m50.334476739s) [noise=true , address=77.90.31.14:4662 , piece_id=KBKIZOXR3PAMMU7SIL66NOZCW6XQCA6C6R4R3WFJ7UTBILSTIYRA, seg_pos={0 0}, upload=507753]: Uploading piece... Dec 29 18:41:50.463446113 (579h12m50.334529158s) [noise=true , address=45.91.116.5:4297 , piece_id=HKJQYPHU4CKAYR23VKJHWP5I4EDFIKHRN3XF3G5JRUDGA7IIYNHQ, seg_pos={0 0}, upload=507753]: Uploading piece... Dec 29 18:41:50.464101852 (579h12m50.335184887s) [noise=true , address=45.91.116.14:4191 , piece_id=IXE7DBDK5MGURE4FMK37C6CTLMHN2EKQOLRWVVIJ7CROAW7KZ7RQ, seg_pos={0 0}, upload=507753]: Uploading piece... Dec 29 18:41:50.464860847 (579h12m50.335943893s) [noise=true , address=145.79.227.31:4798 , piece_id=BMSVTX6CQFGAJQXGZOJOKPW2TMOUWCMXHJ2TM7GJ63U63Q6Y4UHQ, seg_pos={0 0}, upload=507753]: Uploading piece... Dec 29 18:41:50.465471961 (579h12m50.336554996s) [noise=true , address=145.79.227.11:4731 , piece_id=BASLDMVCVW62VRO5GVQ27TAHF2SOKJNSQWV5MLCTLQ6QR72HXQZA, seg_pos={0 0}, upload=507753]: Uploading piece... Dec 29 18:41:50.466352718 (579h12m50.337435763s) [noise=true , address=145.79.227.12:4765 , piece_id=CYADBXI4TA5EJFVOT2C5GQ2VZB5JRM44ODOCYTDXBB7K7K2YAPIQ, seg_pos={0 0}, upload=507753]: Uploading piece... Dec 29 18:41:50.466863611 (579h12m50.337946656s) [noise=true , address=145.79.227.43:4159 , piece_id=IDLZIAMIMX4OUQ725GLAOFPCNBG7LRMK3RK6IOHJDABBQBKDIEHA, seg_pos={0 0}, upload=507753]: Uploading piece... Dec 29 18:41:50.468540144 (579h12m50.339623189s) [noise=true , address=145.79.227.13:4803 , piece_id=Z5UGGQBPCL4WB6EVIO3CXDFTDPZPHC7IJYHHV6R7LRPE4QZZR6HA, seg_pos={0 0}, upload=507753]: Uploading piece... Dec 29 18:41:50.470345731 (579h12m50.341428776s) [noise=true , address=45.91.116.8:4047 , piece_id=3PL4TZ6GHSPHKCDCQJ4CNUCDSVHDKIKUZUDMIZQ3BBBHWN2YCFGQ, seg_pos={0 0}, upload=507753]: Uploading piece... Dec 29 18:41:50.472160918 (579h12m50.343243953s) [noise=true , address=145.79.227.14:4840 , piece_id=ZCONPUMFMFOXCNCNMBI4CKYKTC3VVIPJLASLKHAG77CKUJRJ6L2Q, seg_pos={0 0}, upload=507753]: Uploading piece... Dec 29 18:41:50.474434638 (579h12m50.345517673s) [noise=true , address=145.79.227.42:4123 , piece_id=CWUZK47WYZD5WBP5VP7CW2VXTX5SPSNM47IOVZ6IXR5B3BHMDXWQ, seg_pos={0 0}, upload=507753]: Uploading piece... Dec 29 18:41:50.476533774 (579h12m50.347616819s) [noise=true , address=45.91.116.12:4228 , piece_id=GE6VNK3UVMLSP5GCMTRYJWEMDKALRPXVTEIUA2UY6QPVV6VAEUJA, seg_pos={0 0}, upload=507753]: Uploading piece... Dec 29 18:41:50.477346172 (579h12m50.348429207s) [noise=true , address=145.79.227.52:4447 , piece_id=VXICGUZSIWWT6G43Y6QHI7IYWLHUNUTIC5ULXVEA2IHTXT6S2NSQ, seg_pos={0 0}, upload=507753]: Uploading piece... Dec 29 18:41:50.479081115 (579h12m50.35016415s) [noise=true , address=145.79.227.50:4375 , piece_id=ON7JXLU2VBRC75NC5DU6ZKFEDJ7IQOTAOPWYBC7LPJ5YF4PLVZWA, seg_pos={0 0}, upload=507753]: Uploading piece... Dec 29 18:41:50.479314439 (579h12m50.350397484s) [noise=true , address=145.79.227.16:4912 , piece_id=3OY3DUO3X7CA6B4I7FIQFDLXVWUUD7PGSG4XXXOJ4PVQBVGD65RA, seg_pos={0 0}, upload=507753]: Uploading piece... Dec 29 18:41:52.167275495 (579h12m52.03835854s) [upload=507753]: Next returned nil segment Dec 29 18:41:52.167298408 (579h12m52.038381443s) [upload=507753]: Waiting for error group managing segments... Dec 29 18:41:52.458077026 (579h12m52.329160071s) [noise=true , address=145.79.227.31:4798 , piece_id=BMSVTX6CQFGAJQXGZOJOKPW2TMOUWCMXHJ2TM7GJ63U63Q6Y4UHQ, seg_pos={0 0}, upload=507753]: Done uploading piece. err: <nil> Dec 29 18:41:52.458852071 (579h12m52.329935106s) [noise=true , address=145.79.227.11:4731 , piece_id=BASLDMVCVW62VRO5GVQ27TAHF2SOKJNSQWV5MLCTLQ6QR72HXQZA, seg_pos={0 0}, upload=507753]: Done uploading piece. err: <nil> Dec 29 18:41:52.468901526 (579h12m52.339984571s) [noise=true , address=145.79.227.13:4803 , piece_id=Z5UGGQBPCL4WB6EVIO3CXDFTDPZPHC7IJYHHV6R7LRPE4QZZR6HA, seg_pos={0 0}, upload=507753]: Done uploading piece. err: <nil> Dec 29 18:41:52.473572330 (579h12m52.344655365s) [noise=true , address=145.79.227.16:4912 , piece_id=3OY3DUO3X7CA6B4I7FIQFDLXVWUUD7PGSG4XXXOJ4PVQBVGD65RA, seg_pos={0 0}, upload=507753]: Done uploading piece. err: <nil> Dec 29 18:41:52.481128078 (579h12m52.352211113s) [noise=true , address=145.79.227.50:4375 , piece_id=ON7JXLU2VBRC75NC5DU6ZKFEDJ7IQOTAOPWYBC7LPJ5YF4PLVZWA, seg_pos={0 0}, upload=507753]: Done uploading piece. err: <nil> Dec 29 18:41:52.481889758 (579h12m52.352972793s) [noise=true , address=145.79.227.12:4765 , piece_id=CYADBXI4TA5EJFVOT2C5GQ2VZB5JRM44ODOCYTDXBB7K7K2YAPIQ, seg_pos={0 0}, upload=507753]: Done uploading piece. err: <nil> Dec 29 18:41:52.484012891 (579h12m52.355095926s) [noise=true , address=145.79.227.42:4123 , piece_id=CWUZK47WYZD5WBP5VP7CW2VXTX5SPSNM47IOVZ6IXR5B3BHMDXWQ, seg_pos={0 0}, upload=507753]: Done uploading piece. err: <nil> Dec 29 18:41:52.493568666 (579h12m52.364651701s) [noise=true , address=145.79.227.14:4840 , piece_id=ZCONPUMFMFOXCNCNMBI4CKYKTC3VVIPJLASLKHAG77CKUJRJ6L2Q, seg_pos={0 0}, upload=507753]: Done uploading piece. err: <nil> Dec 29 18:41:52.498593054 (579h12m52.369676089s) [noise=true , address=145.79.227.52:4447 , piece_id=VXICGUZSIWWT6G43Y6QHI7IYWLHUNUTIC5ULXVEA2IHTXT6S2NSQ, seg_pos={0 0}, upload=507753]: Done uploading piece. err: <nil> Dec 29 18:41:52.659063574 (579h12m52.530146619s) [noise=true , address=77.90.31.14:4662 , piece_id=KBKIZOXR3PAMMU7SIL66NOZCW6XQCA6C6R4R3WFJ7UTBILSTIYRA, seg_pos={0 0}, upload=507753]: Done uploading piece. err: <nil> Dec 29 18:41:52.683381218 (579h12m52.554464263s) [noise=true , address=45.91.116.5:4297 , piece_id=HKJQYPHU4CKAYR23VKJHWP5I4EDFIKHRN3XF3G5JRUDGA7IIYNHQ, seg_pos={0 0}, upload=507753]: Done uploading piece. err: <nil> Dec 29 18:41:52.694320649 (579h12m52.565403684s) [noise=true , address=77.90.31.18:4808 , piece_id=O2APWAJ7OUPHWV6W2MOETRITDK7SA5CT5HLCZNCQRXVDXN6TEM4A, seg_pos={0 0}, upload=507753]: Done uploading piece. err: <nil> Dec 29 18:41:52.724640933 (579h12m52.595723968s) [noise=true , address=45.91.116.14:4191 , piece_id=IXE7DBDK5MGURE4FMK37C6CTLMHN2EKQOLRWVVIJ7CROAW7KZ7RQ, seg_pos={0 0}, upload=507753]: Done uploading piece. err: <nil> Dec 29 18:41:52.724665169 (579h12m52.595748204s) [seg_pos={0 0}, upload=507753]: Segment reached optimal threshold of 13 pieces. Dec 29 18:41:52.724903403 (579h12m52.595986448s) [noise=true , address=45.91.116.12:4228 , piece_id=GE6VNK3UVMLSP5GCMTRYJWEMDKALRPXVTEIUA2UY6QPVV6VAEUJA, seg_pos={0 0}, upload=507753]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 29 18:41:52.724915235 (579h12m52.59599828s) [noise=true , address=45.91.116.12:4228 , piece_id=GE6VNK3UVMLSP5GCMTRYJWEMDKALRPXVTEIUA2UY6QPVV6VAEUJA, seg_pos={0 0}, upload=507753]: Long tail cancellation due to optimal threshold Dec 29 18:41:52.725238331 (579h12m52.596321376s) [noise=true , address=145.79.227.43:4159 , piece_id=IDLZIAMIMX4OUQ725GLAOFPCNBG7LRMK3RK6IOHJDABBQBKDIEHA, seg_pos={0 0}, upload=507753]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 29 18:41:52.725246577 (579h12m52.596329622s) [noise=true , address=145.79.227.43:4159 , piece_id=IDLZIAMIMX4OUQ725GLAOFPCNBG7LRMK3RK6IOHJDABBQBKDIEHA, seg_pos={0 0}, upload=507753]: Long tail cancellation due to optimal threshold Dec 29 18:41:52.725408354 (579h12m52.596491399s) [noise=true , address=45.91.116.8:4047 , piece_id=3PL4TZ6GHSPHKCDCQJ4CNUCDSVHDKIKUZUDMIZQ3BBBHWN2YCFGQ, seg_pos={0 0}, upload=507753]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:37536->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:37536->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37536->45.91.116.8:4047: use of closed network connection Dec 29 18:41:52.725416901 (579h12m52.596499936s) [noise=true , address=45.91.116.8:4047 , piece_id=3PL4TZ6GHSPHKCDCQJ4CNUCDSVHDKIKUZUDMIZQ3BBBHWN2YCFGQ, seg_pos={0 0}, upload=507753]: Long tail cancellation due to optimal threshold Dec 29 18:41:52.725436949 (579h12m52.596519994s) [seg_pos={0 0}, upload=507753]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:41:52.725522482 (579h12m52.596605517s) [upload=507753]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:41:52.725528644 (579h12m52.596611689s) [upload=507753]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:41:53.015240038 (579h12m52.886323083s) [upload=507753]: Done uploading segments... Dec 29 18:41:53.015268191 (579h12m52.886351237s) [upload=507753]: Upload finished. err: <nil> Duration: 30.003770587s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 3.530089284s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:42:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.530113632s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 27.341021093s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:41:44 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 27.34105069s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X..X.XXX..X.....X.X.....X...XX.X......XXX.XX.X.XX.......X..] duration: 5.959283283s trace: https://jaeger.dev.storj.io/trace/205c371f3e3d99d5 FAIL (2025-12-29 18:41:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: ZSVNVESQCLSYSELZABTYN5CUYNFRCSEI2OJVRIWUDAGJOBGWLFPA) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: QNCDEGLPQ2HQUV3ZJWTKLOFZT2OQ5JK67JQEAJDJGU7GNZAC5WBA) context deadline exceeded ======LOGS====== Dec 29 18:39:05.269762644 (579h10m5.140845679s) [upload=507714]: Starting upload Dec 29 18:39:05.269842036 (579h10m5.140925081s) [upload=507714]: Done starting upload Dec 29 18:39:05.270022539 (579h10m5.141105574s) [upload=507714]: Uploading segments... Dec 29 18:39:05.270072644 (579h10m5.141155689s) [upload=507714]: Got next segment. Inline: false Dec 29 18:39:05.270102702 (579h10m5.141185747s) [upload=507714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:39:05.270108883 (579h10m5.141191928s) [upload=507714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:39:05.516713306 (579h10m5.387796351s) [seg_pos={0 0}, upload=507714]: Begin upload segment... Dec 29 18:39:05.516761808 (579h10m5.387844853s) [seg_pos={0 0}, upload=507714]: Done begin upload segment. Dec 29 18:39:05.516793349 (579h10m5.387876384s) [noise=true , address=45.91.116.11:4117 , piece_id=QNCDEGLPQ2HQUV3ZJWTKLOFZT2OQ5JK67JQEAJDJGU7GNZAC5WBA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.516801073 (579h10m5.387884108s) [noise=true , address=77.90.31.11:4554 , piece_id=ZSVNVESQCLSYSELZABTYN5CUYNFRCSEI2OJVRIWUDAGJOBGWLFPA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.516848785 (579h10m5.38793182s) [noise=true , address=77.90.31.19:4841 , piece_id=JUORST5Y4RC4DQ66SWYCTBVWDYQVUL5FYAK6AEDXZKPROKIMYYGA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.517226514 (579h10m5.388309559s) [noise=true , address=145.79.227.47:4267 , piece_id=46JT7SHEI3AZZJFE3WAN7C7Q2LLHUEJPKPQV4AUIVMY3PD7HJSHQ, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.517508631 (579h10m5.388591676s) [noise=true , address=77.90.31.14:4664 , piece_id=Q2BNXSNP2ODUYDXYG7JGWQLLCCJRYV5LU3YRR2LLWD2RUGZLQV5Q, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.517767684 (579h10m5.388850729s) [noise=true , address=145.79.227.28:4725 , piece_id=QIYZ7MCB4PIAAWVGRBLP7OVY3RAYG6KKXRTCTJQN7RMJY7P324AQ, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.518158778 (579h10m5.389241813s) [noise=true , address=145.79.227.34:4907 , piece_id=U75CXZ7KXGIH2ZBJCUXS5XBLR4KTH5OF7RTX2M7FDRPZZRLICLYQ, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.518496552 (579h10m5.389579597s) [noise=true , address=45.91.116.10:4082 , piece_id=37KJTWGAFQ2HLDYOQDKPLYHI45KEV65SWFTPKE4ACXPZSHE5JSIA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.518910821 (579h10m5.389993866s) [noise=true , address=145.79.227.19:4079 , piece_id=LYABMAFFM4NN3LF37ORRYJXH3C4OYB6JVQNSX574HHJ673BLNU2Q, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.519333395 (579h10m5.39041643s) [noise=true , address=145.79.227.11:4731 , piece_id=WH3EENCJGWCYVOGVJHJIYDVPOSM3IDML2YQ6FUNGTRJLVXNRKF6A, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.519765489 (579h10m5.390848534s) [noise=true , address=145.79.227.10:4402 , piece_id=R5NGJHILXHOEUXUEDGJ6NCJVNZGLPYSQ6NQQZRND6KXQ3SZFWAQA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.520534152 (579h10m5.391617187s) [noise=true , address=77.90.31.16:4735 , piece_id=FO45RZOCNBD37N3RCNKCUQXZK4QKMRXLTLX5VRMQENMZDMLMTLDA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.520957829 (579h10m5.392040874s) [noise=true , address=45.91.116.12:4227 , piece_id=ZDTAVEVZ4Y6QWAM5WQU47QCDI4HBNRJ2EZ66NKUW2G7M6PT7QJYA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.521587858 (579h10m5.392670903s) [noise=true , address=145.79.227.33:4871 , piece_id=BFWOZNIPIDIRAA7XZJIO5U64JFT2KMAXFTQCGX4G3BBYET7KJFUQ, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.523484429 (579h10m5.394567464s) [noise=true , address=145.79.227.50:4375 , piece_id=URQT6EIJJT4ISQQ2LO4MBGPOFB45LASM7JP6IW63A5PWLOFDS6BA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.524676169 (579h10m5.395759214s) [noise=true , address=45.91.116.7:4010 , piece_id=7GFKGTUGNF544HFIXIFBOMMQBH34KHMF2B2ED6UAV2OXWC6F4A5Q, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:07.233782657 (579h10m7.104865692s) [upload=507714]: Next returned nil segment Dec 29 18:39:07.233825318 (579h10m7.104908353s) [upload=507714]: Waiting for error group managing segments... Dec 29 18:39:07.521013829 (579h10m7.392096874s) [noise=true , address=145.79.227.10:4402 , piece_id=R5NGJHILXHOEUXUEDGJ6NCJVNZGLPYSQ6NQQZRND6KXQ3SZFWAQA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.521890689 (579h10m7.392973734s) [noise=true , address=145.79.227.19:4079 , piece_id=LYABMAFFM4NN3LF37ORRYJXH3C4OYB6JVQNSX574HHJ673BLNU2Q, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.525031880 (579h10m7.396114925s) [noise=true , address=145.79.227.28:4725 , piece_id=QIYZ7MCB4PIAAWVGRBLP7OVY3RAYG6KKXRTCTJQN7RMJY7P324AQ, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.543925366 (579h10m7.415008411s) [noise=true , address=145.79.227.47:4267 , piece_id=46JT7SHEI3AZZJFE3WAN7C7Q2LLHUEJPKPQV4AUIVMY3PD7HJSHQ, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.559780046 (579h10m7.430863091s) [noise=true , address=145.79.227.11:4731 , piece_id=WH3EENCJGWCYVOGVJHJIYDVPOSM3IDML2YQ6FUNGTRJLVXNRKF6A, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.740401066 (579h10m7.611484101s) [noise=true , address=45.91.116.11:4117 , piece_id=QNCDEGLPQ2HQUV3ZJWTKLOFZT2OQ5JK67JQEAJDJGU7GNZAC5WBA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.853332184 (579h10m7.724415229s) [noise=true , address=45.91.116.12:4227 , piece_id=ZDTAVEVZ4Y6QWAM5WQU47QCDI4HBNRJ2EZ66NKUW2G7M6PT7QJYA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.897665411 (579h10m7.768748476s) [noise=true , address=145.79.227.50:4375 , piece_id=URQT6EIJJT4ISQQ2LO4MBGPOFB45LASM7JP6IW63A5PWLOFDS6BA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.951662390 (579h10m7.822745435s) [noise=true , address=45.91.116.7:4010 , piece_id=7GFKGTUGNF544HFIXIFBOMMQBH34KHMF2B2ED6UAV2OXWC6F4A5Q, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:08.506526966 (579h10m8.377610011s) [noise=true , address=45.91.116.10:4082 , piece_id=37KJTWGAFQ2HLDYOQDKPLYHI45KEV65SWFTPKE4ACXPZSHE5JSIA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:08.843235771 (579h10m8.714318806s) [noise=true , address=145.79.227.34:4907 , piece_id=U75CXZ7KXGIH2ZBJCUXS5XBLR4KTH5OF7RTX2M7FDRPZZRLICLYQ, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:09.174697294 (579h10m9.045780339s) [noise=true , address=145.79.227.33:4871 , piece_id=BFWOZNIPIDIRAA7XZJIO5U64JFT2KMAXFTQCGX4G3BBYET7KJFUQ, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:10.635844411 (579h10m10.506927456s) [noise=true , address=77.90.31.11:4554 , piece_id=ZSVNVESQCLSYSELZABTYN5CUYNFRCSEI2OJVRIWUDAGJOBGWLFPA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:10.635868427 (579h10m10.506951472s) [seg_pos={0 0}, upload=507714]: Segment reached optimal threshold of 13 pieces. Dec 29 18:39:10.636079248 (579h10m10.507162283s) [noise=true , address=77.90.31.14:4664 , piece_id=Q2BNXSNP2ODUYDXYG7JGWQLLCCJRYV5LU3YRR2LLWD2RUGZLQV5Q, seg_pos={0 0}, upload=507714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 29 18:39:10.636086802 (579h10m10.507169848s) [noise=true , address=77.90.31.14:4664 , piece_id=Q2BNXSNP2ODUYDXYG7JGWQLLCCJRYV5LU3YRR2LLWD2RUGZLQV5Q, seg_pos={0 0}, upload=507714]: Long tail cancellation due to optimal threshold Dec 29 18:39:10.636202763 (579h10m10.507285808s) [noise=true , address=77.90.31.16:4735 , piece_id=FO45RZOCNBD37N3RCNKCUQXZK4QKMRXLTLX5VRMQENMZDMLMTLDA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 29 18:39:10.636223914 (579h10m10.507306949s) [noise=true , address=77.90.31.16:4735 , piece_id=FO45RZOCNBD37N3RCNKCUQXZK4QKMRXLTLX5VRMQENMZDMLMTLDA, seg_pos={0 0}, upload=507714]: Long tail cancellation due to optimal threshold Dec 29 18:39:10.636256365 (579h10m10.507339401s) [noise=true , address=77.90.31.19:4841 , piece_id=JUORST5Y4RC4DQ66SWYCTBVWDYQVUL5FYAK6AEDXZKPROKIMYYGA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Dec 29 18:39:10.636264411 (579h10m10.507347456s) [noise=true , address=77.90.31.19:4841 , piece_id=JUORST5Y4RC4DQ66SWYCTBVWDYQVUL5FYAK6AEDXZKPROKIMYYGA, seg_pos={0 0}, upload=507714]: Long tail cancellation due to optimal threshold Dec 29 18:39:10.636291773 (579h10m10.507385538s) [seg_pos={0 0}, upload=507714]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:39:10.636416190 (579h10m10.507499225s) [upload=507714]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:39:10.636422712 (579h10m10.507505748s) [upload=507714]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:39:10.932641628 (579h10m10.803724673s) [upload=507714]: Done uploading segments... Dec 29 18:39:10.932664472 (579h10m10.803747507s) [upload=507714]: Upload finished. err: <nil> Duration: 5.959315134s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 26.575404575s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:40:59 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 26.575432689s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 3.516992621s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:40:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.517021045s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 3.4150689s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:40:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.415094628s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X.X..X.XXX..X.....X.X.....X...XX.X......XXX.XX.X.XX.......X.] duration: 6.090212775s trace: https://jaeger.dev.storj.io/trace/205c371f3e3d99d5 FAIL (2025-12-29 18:40:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: ZSVNVESQCLSYSELZABTYN5CUYNFRCSEI2OJVRIWUDAGJOBGWLFPA) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: QNCDEGLPQ2HQUV3ZJWTKLOFZT2OQ5JK67JQEAJDJGU7GNZAC5WBA) context deadline exceeded ======LOGS====== Dec 29 18:39:05.269762644 (579h10m5.140845679s) [upload=507714]: Starting upload Dec 29 18:39:05.269842036 (579h10m5.140925081s) [upload=507714]: Done starting upload Dec 29 18:39:05.270022539 (579h10m5.141105574s) [upload=507714]: Uploading segments... Dec 29 18:39:05.270072644 (579h10m5.141155689s) [upload=507714]: Got next segment. Inline: false Dec 29 18:39:05.270102702 (579h10m5.141185747s) [upload=507714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:39:05.270108883 (579h10m5.141191928s) [upload=507714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:39:05.516713306 (579h10m5.387796351s) [seg_pos={0 0}, upload=507714]: Begin upload segment... Dec 29 18:39:05.516761808 (579h10m5.387844853s) [seg_pos={0 0}, upload=507714]: Done begin upload segment. Dec 29 18:39:05.516793349 (579h10m5.387876384s) [noise=true , address=45.91.116.11:4117 , piece_id=QNCDEGLPQ2HQUV3ZJWTKLOFZT2OQ5JK67JQEAJDJGU7GNZAC5WBA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.516801073 (579h10m5.387884108s) [noise=true , address=77.90.31.11:4554 , piece_id=ZSVNVESQCLSYSELZABTYN5CUYNFRCSEI2OJVRIWUDAGJOBGWLFPA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.516848785 (579h10m5.38793182s) [noise=true , address=77.90.31.19:4841 , piece_id=JUORST5Y4RC4DQ66SWYCTBVWDYQVUL5FYAK6AEDXZKPROKIMYYGA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.517226514 (579h10m5.388309559s) [noise=true , address=145.79.227.47:4267 , piece_id=46JT7SHEI3AZZJFE3WAN7C7Q2LLHUEJPKPQV4AUIVMY3PD7HJSHQ, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.517508631 (579h10m5.388591676s) [noise=true , address=77.90.31.14:4664 , piece_id=Q2BNXSNP2ODUYDXYG7JGWQLLCCJRYV5LU3YRR2LLWD2RUGZLQV5Q, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.517767684 (579h10m5.388850729s) [noise=true , address=145.79.227.28:4725 , piece_id=QIYZ7MCB4PIAAWVGRBLP7OVY3RAYG6KKXRTCTJQN7RMJY7P324AQ, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.518158778 (579h10m5.389241813s) [noise=true , address=145.79.227.34:4907 , piece_id=U75CXZ7KXGIH2ZBJCUXS5XBLR4KTH5OF7RTX2M7FDRPZZRLICLYQ, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.518496552 (579h10m5.389579597s) [noise=true , address=45.91.116.10:4082 , piece_id=37KJTWGAFQ2HLDYOQDKPLYHI45KEV65SWFTPKE4ACXPZSHE5JSIA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.518910821 (579h10m5.389993866s) [noise=true , address=145.79.227.19:4079 , piece_id=LYABMAFFM4NN3LF37ORRYJXH3C4OYB6JVQNSX574HHJ673BLNU2Q, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.519333395 (579h10m5.39041643s) [noise=true , address=145.79.227.11:4731 , piece_id=WH3EENCJGWCYVOGVJHJIYDVPOSM3IDML2YQ6FUNGTRJLVXNRKF6A, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.519765489 (579h10m5.390848534s) [noise=true , address=145.79.227.10:4402 , piece_id=R5NGJHILXHOEUXUEDGJ6NCJVNZGLPYSQ6NQQZRND6KXQ3SZFWAQA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.520534152 (579h10m5.391617187s) [noise=true , address=77.90.31.16:4735 , piece_id=FO45RZOCNBD37N3RCNKCUQXZK4QKMRXLTLX5VRMQENMZDMLMTLDA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.520957829 (579h10m5.392040874s) [noise=true , address=45.91.116.12:4227 , piece_id=ZDTAVEVZ4Y6QWAM5WQU47QCDI4HBNRJ2EZ66NKUW2G7M6PT7QJYA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.521587858 (579h10m5.392670903s) [noise=true , address=145.79.227.33:4871 , piece_id=BFWOZNIPIDIRAA7XZJIO5U64JFT2KMAXFTQCGX4G3BBYET7KJFUQ, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.523484429 (579h10m5.394567464s) [noise=true , address=145.79.227.50:4375 , piece_id=URQT6EIJJT4ISQQ2LO4MBGPOFB45LASM7JP6IW63A5PWLOFDS6BA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.524676169 (579h10m5.395759214s) [noise=true , address=45.91.116.7:4010 , piece_id=7GFKGTUGNF544HFIXIFBOMMQBH34KHMF2B2ED6UAV2OXWC6F4A5Q, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:07.233782657 (579h10m7.104865692s) [upload=507714]: Next returned nil segment Dec 29 18:39:07.233825318 (579h10m7.104908353s) [upload=507714]: Waiting for error group managing segments... Dec 29 18:39:07.521013829 (579h10m7.392096874s) [noise=true , address=145.79.227.10:4402 , piece_id=R5NGJHILXHOEUXUEDGJ6NCJVNZGLPYSQ6NQQZRND6KXQ3SZFWAQA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.521890689 (579h10m7.392973734s) [noise=true , address=145.79.227.19:4079 , piece_id=LYABMAFFM4NN3LF37ORRYJXH3C4OYB6JVQNSX574HHJ673BLNU2Q, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.525031880 (579h10m7.396114925s) [noise=true , address=145.79.227.28:4725 , piece_id=QIYZ7MCB4PIAAWVGRBLP7OVY3RAYG6KKXRTCTJQN7RMJY7P324AQ, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.543925366 (579h10m7.415008411s) [noise=true , address=145.79.227.47:4267 , piece_id=46JT7SHEI3AZZJFE3WAN7C7Q2LLHUEJPKPQV4AUIVMY3PD7HJSHQ, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.559780046 (579h10m7.430863091s) [noise=true , address=145.79.227.11:4731 , piece_id=WH3EENCJGWCYVOGVJHJIYDVPOSM3IDML2YQ6FUNGTRJLVXNRKF6A, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.740401066 (579h10m7.611484101s) [noise=true , address=45.91.116.11:4117 , piece_id=QNCDEGLPQ2HQUV3ZJWTKLOFZT2OQ5JK67JQEAJDJGU7GNZAC5WBA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.853332184 (579h10m7.724415229s) [noise=true , address=45.91.116.12:4227 , piece_id=ZDTAVEVZ4Y6QWAM5WQU47QCDI4HBNRJ2EZ66NKUW2G7M6PT7QJYA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.897665411 (579h10m7.768748476s) [noise=true , address=145.79.227.50:4375 , piece_id=URQT6EIJJT4ISQQ2LO4MBGPOFB45LASM7JP6IW63A5PWLOFDS6BA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.951662390 (579h10m7.822745435s) [noise=true , address=45.91.116.7:4010 , piece_id=7GFKGTUGNF544HFIXIFBOMMQBH34KHMF2B2ED6UAV2OXWC6F4A5Q, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:08.506526966 (579h10m8.377610011s) [noise=true , address=45.91.116.10:4082 , piece_id=37KJTWGAFQ2HLDYOQDKPLYHI45KEV65SWFTPKE4ACXPZSHE5JSIA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:08.843235771 (579h10m8.714318806s) [noise=true , address=145.79.227.34:4907 , piece_id=U75CXZ7KXGIH2ZBJCUXS5XBLR4KTH5OF7RTX2M7FDRPZZRLICLYQ, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:09.174697294 (579h10m9.045780339s) [noise=true , address=145.79.227.33:4871 , piece_id=BFWOZNIPIDIRAA7XZJIO5U64JFT2KMAXFTQCGX4G3BBYET7KJFUQ, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:10.635844411 (579h10m10.506927456s) [noise=true , address=77.90.31.11:4554 , piece_id=ZSVNVESQCLSYSELZABTYN5CUYNFRCSEI2OJVRIWUDAGJOBGWLFPA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:10.635868427 (579h10m10.506951472s) [seg_pos={0 0}, upload=507714]: Segment reached optimal threshold of 13 pieces. Dec 29 18:39:10.636079248 (579h10m10.507162283s) [noise=true , address=77.90.31.14:4664 , piece_id=Q2BNXSNP2ODUYDXYG7JGWQLLCCJRYV5LU3YRR2LLWD2RUGZLQV5Q, seg_pos={0 0}, upload=507714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 29 18:39:10.636086802 (579h10m10.507169848s) [noise=true , address=77.90.31.14:4664 , piece_id=Q2BNXSNP2ODUYDXYG7JGWQLLCCJRYV5LU3YRR2LLWD2RUGZLQV5Q, seg_pos={0 0}, upload=507714]: Long tail cancellation due to optimal threshold Dec 29 18:39:10.636202763 (579h10m10.507285808s) [noise=true , address=77.90.31.16:4735 , piece_id=FO45RZOCNBD37N3RCNKCUQXZK4QKMRXLTLX5VRMQENMZDMLMTLDA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 29 18:39:10.636223914 (579h10m10.507306949s) [noise=true , address=77.90.31.16:4735 , piece_id=FO45RZOCNBD37N3RCNKCUQXZK4QKMRXLTLX5VRMQENMZDMLMTLDA, seg_pos={0 0}, upload=507714]: Long tail cancellation due to optimal threshold Dec 29 18:39:10.636256365 (579h10m10.507339401s) [noise=true , address=77.90.31.19:4841 , piece_id=JUORST5Y4RC4DQ66SWYCTBVWDYQVUL5FYAK6AEDXZKPROKIMYYGA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Dec 29 18:39:10.636264411 (579h10m10.507347456s) [noise=true , address=77.90.31.19:4841 , piece_id=JUORST5Y4RC4DQ66SWYCTBVWDYQVUL5FYAK6AEDXZKPROKIMYYGA, seg_pos={0 0}, upload=507714]: Long tail cancellation due to optimal threshold Dec 29 18:39:10.636291773 (579h10m10.507385538s) [seg_pos={0 0}, upload=507714]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:39:10.636416190 (579h10m10.507499225s) [upload=507714]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:39:10.636422712 (579h10m10.507505748s) [upload=507714]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:39:10.932641628 (579h10m10.803724673s) [upload=507714]: Done uploading segments... Dec 29 18:39:10.932664472 (579h10m10.803747507s) [upload=507714]: Upload finished. err: <nil> Duration: 6.090244705s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 3.479477648s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:40:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.479632282s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 15.41220763s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:39:39 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 15.412237277s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X.X..X.XXX..X.....X.X.....X...XX.X......XXX.XX.X.XX.......X] duration: 30.003213554s trace: https://jaeger.dev.storj.io/trace/205c371f3e3d99d5 FAIL (2025-12-29 18:39:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: ZSVNVESQCLSYSELZABTYN5CUYNFRCSEI2OJVRIWUDAGJOBGWLFPA) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: QNCDEGLPQ2HQUV3ZJWTKLOFZT2OQ5JK67JQEAJDJGU7GNZAC5WBA) context deadline exceeded ======LOGS====== Dec 29 18:39:05.269762644 (579h10m5.140845679s) [upload=507714]: Starting upload Dec 29 18:39:05.269842036 (579h10m5.140925081s) [upload=507714]: Done starting upload Dec 29 18:39:05.270022539 (579h10m5.141105574s) [upload=507714]: Uploading segments... Dec 29 18:39:05.270072644 (579h10m5.141155689s) [upload=507714]: Got next segment. Inline: false Dec 29 18:39:05.270102702 (579h10m5.141185747s) [upload=507714]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:39:05.270108883 (579h10m5.141191928s) [upload=507714]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:39:05.516713306 (579h10m5.387796351s) [seg_pos={0 0}, upload=507714]: Begin upload segment... Dec 29 18:39:05.516761808 (579h10m5.387844853s) [seg_pos={0 0}, upload=507714]: Done begin upload segment. Dec 29 18:39:05.516793349 (579h10m5.387876384s) [noise=true , address=45.91.116.11:4117 , piece_id=QNCDEGLPQ2HQUV3ZJWTKLOFZT2OQ5JK67JQEAJDJGU7GNZAC5WBA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.516801073 (579h10m5.387884108s) [noise=true , address=77.90.31.11:4554 , piece_id=ZSVNVESQCLSYSELZABTYN5CUYNFRCSEI2OJVRIWUDAGJOBGWLFPA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.516848785 (579h10m5.38793182s) [noise=true , address=77.90.31.19:4841 , piece_id=JUORST5Y4RC4DQ66SWYCTBVWDYQVUL5FYAK6AEDXZKPROKIMYYGA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.517226514 (579h10m5.388309559s) [noise=true , address=145.79.227.47:4267 , piece_id=46JT7SHEI3AZZJFE3WAN7C7Q2LLHUEJPKPQV4AUIVMY3PD7HJSHQ, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.517508631 (579h10m5.388591676s) [noise=true , address=77.90.31.14:4664 , piece_id=Q2BNXSNP2ODUYDXYG7JGWQLLCCJRYV5LU3YRR2LLWD2RUGZLQV5Q, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.517767684 (579h10m5.388850729s) [noise=true , address=145.79.227.28:4725 , piece_id=QIYZ7MCB4PIAAWVGRBLP7OVY3RAYG6KKXRTCTJQN7RMJY7P324AQ, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.518158778 (579h10m5.389241813s) [noise=true , address=145.79.227.34:4907 , piece_id=U75CXZ7KXGIH2ZBJCUXS5XBLR4KTH5OF7RTX2M7FDRPZZRLICLYQ, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.518496552 (579h10m5.389579597s) [noise=true , address=45.91.116.10:4082 , piece_id=37KJTWGAFQ2HLDYOQDKPLYHI45KEV65SWFTPKE4ACXPZSHE5JSIA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.518910821 (579h10m5.389993866s) [noise=true , address=145.79.227.19:4079 , piece_id=LYABMAFFM4NN3LF37ORRYJXH3C4OYB6JVQNSX574HHJ673BLNU2Q, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.519333395 (579h10m5.39041643s) [noise=true , address=145.79.227.11:4731 , piece_id=WH3EENCJGWCYVOGVJHJIYDVPOSM3IDML2YQ6FUNGTRJLVXNRKF6A, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.519765489 (579h10m5.390848534s) [noise=true , address=145.79.227.10:4402 , piece_id=R5NGJHILXHOEUXUEDGJ6NCJVNZGLPYSQ6NQQZRND6KXQ3SZFWAQA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.520534152 (579h10m5.391617187s) [noise=true , address=77.90.31.16:4735 , piece_id=FO45RZOCNBD37N3RCNKCUQXZK4QKMRXLTLX5VRMQENMZDMLMTLDA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.520957829 (579h10m5.392040874s) [noise=true , address=45.91.116.12:4227 , piece_id=ZDTAVEVZ4Y6QWAM5WQU47QCDI4HBNRJ2EZ66NKUW2G7M6PT7QJYA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.521587858 (579h10m5.392670903s) [noise=true , address=145.79.227.33:4871 , piece_id=BFWOZNIPIDIRAA7XZJIO5U64JFT2KMAXFTQCGX4G3BBYET7KJFUQ, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.523484429 (579h10m5.394567464s) [noise=true , address=145.79.227.50:4375 , piece_id=URQT6EIJJT4ISQQ2LO4MBGPOFB45LASM7JP6IW63A5PWLOFDS6BA, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:05.524676169 (579h10m5.395759214s) [noise=true , address=45.91.116.7:4010 , piece_id=7GFKGTUGNF544HFIXIFBOMMQBH34KHMF2B2ED6UAV2OXWC6F4A5Q, seg_pos={0 0}, upload=507714]: Uploading piece... Dec 29 18:39:07.233782657 (579h10m7.104865692s) [upload=507714]: Next returned nil segment Dec 29 18:39:07.233825318 (579h10m7.104908353s) [upload=507714]: Waiting for error group managing segments... Dec 29 18:39:07.521013829 (579h10m7.392096874s) [noise=true , address=145.79.227.10:4402 , piece_id=R5NGJHILXHOEUXUEDGJ6NCJVNZGLPYSQ6NQQZRND6KXQ3SZFWAQA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.521890689 (579h10m7.392973734s) [noise=true , address=145.79.227.19:4079 , piece_id=LYABMAFFM4NN3LF37ORRYJXH3C4OYB6JVQNSX574HHJ673BLNU2Q, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.525031880 (579h10m7.396114925s) [noise=true , address=145.79.227.28:4725 , piece_id=QIYZ7MCB4PIAAWVGRBLP7OVY3RAYG6KKXRTCTJQN7RMJY7P324AQ, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.543925366 (579h10m7.415008411s) [noise=true , address=145.79.227.47:4267 , piece_id=46JT7SHEI3AZZJFE3WAN7C7Q2LLHUEJPKPQV4AUIVMY3PD7HJSHQ, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.559780046 (579h10m7.430863091s) [noise=true , address=145.79.227.11:4731 , piece_id=WH3EENCJGWCYVOGVJHJIYDVPOSM3IDML2YQ6FUNGTRJLVXNRKF6A, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.740401066 (579h10m7.611484101s) [noise=true , address=45.91.116.11:4117 , piece_id=QNCDEGLPQ2HQUV3ZJWTKLOFZT2OQ5JK67JQEAJDJGU7GNZAC5WBA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.853332184 (579h10m7.724415229s) [noise=true , address=45.91.116.12:4227 , piece_id=ZDTAVEVZ4Y6QWAM5WQU47QCDI4HBNRJ2EZ66NKUW2G7M6PT7QJYA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.897665411 (579h10m7.768748476s) [noise=true , address=145.79.227.50:4375 , piece_id=URQT6EIJJT4ISQQ2LO4MBGPOFB45LASM7JP6IW63A5PWLOFDS6BA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:07.951662390 (579h10m7.822745435s) [noise=true , address=45.91.116.7:4010 , piece_id=7GFKGTUGNF544HFIXIFBOMMQBH34KHMF2B2ED6UAV2OXWC6F4A5Q, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:08.506526966 (579h10m8.377610011s) [noise=true , address=45.91.116.10:4082 , piece_id=37KJTWGAFQ2HLDYOQDKPLYHI45KEV65SWFTPKE4ACXPZSHE5JSIA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:08.843235771 (579h10m8.714318806s) [noise=true , address=145.79.227.34:4907 , piece_id=U75CXZ7KXGIH2ZBJCUXS5XBLR4KTH5OF7RTX2M7FDRPZZRLICLYQ, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:09.174697294 (579h10m9.045780339s) [noise=true , address=145.79.227.33:4871 , piece_id=BFWOZNIPIDIRAA7XZJIO5U64JFT2KMAXFTQCGX4G3BBYET7KJFUQ, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:10.635844411 (579h10m10.506927456s) [noise=true , address=77.90.31.11:4554 , piece_id=ZSVNVESQCLSYSELZABTYN5CUYNFRCSEI2OJVRIWUDAGJOBGWLFPA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: <nil> Dec 29 18:39:10.635868427 (579h10m10.506951472s) [seg_pos={0 0}, upload=507714]: Segment reached optimal threshold of 13 pieces. Dec 29 18:39:10.636079248 (579h10m10.507162283s) [noise=true , address=77.90.31.14:4664 , piece_id=Q2BNXSNP2ODUYDXYG7JGWQLLCCJRYV5LU3YRR2LLWD2RUGZLQV5Q, seg_pos={0 0}, upload=507714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 29 18:39:10.636086802 (579h10m10.507169848s) [noise=true , address=77.90.31.14:4664 , piece_id=Q2BNXSNP2ODUYDXYG7JGWQLLCCJRYV5LU3YRR2LLWD2RUGZLQV5Q, seg_pos={0 0}, upload=507714]: Long tail cancellation due to optimal threshold Dec 29 18:39:10.636202763 (579h10m10.507285808s) [noise=true , address=77.90.31.16:4735 , piece_id=FO45RZOCNBD37N3RCNKCUQXZK4QKMRXLTLX5VRMQENMZDMLMTLDA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 29 18:39:10.636223914 (579h10m10.507306949s) [noise=true , address=77.90.31.16:4735 , piece_id=FO45RZOCNBD37N3RCNKCUQXZK4QKMRXLTLX5VRMQENMZDMLMTLDA, seg_pos={0 0}, upload=507714]: Long tail cancellation due to optimal threshold Dec 29 18:39:10.636256365 (579h10m10.507339401s) [noise=true , address=77.90.31.19:4841 , piece_id=JUORST5Y4RC4DQ66SWYCTBVWDYQVUL5FYAK6AEDXZKPROKIMYYGA, seg_pos={0 0}, upload=507714]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Dec 29 18:39:10.636264411 (579h10m10.507347456s) [noise=true , address=77.90.31.19:4841 , piece_id=JUORST5Y4RC4DQ66SWYCTBVWDYQVUL5FYAK6AEDXZKPROKIMYYGA, seg_pos={0 0}, upload=507714]: Long tail cancellation due to optimal threshold Dec 29 18:39:10.636291773 (579h10m10.507385538s) [seg_pos={0 0}, upload=507714]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:39:10.636416190 (579h10m10.507499225s) [upload=507714]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:39:10.636422712 (579h10m10.507505748s) [upload=507714]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:39:10.932641628 (579h10m10.803724673s) [upload=507714]: Done uploading segments... Dec 29 18:39:10.932664472 (579h10m10.803747507s) [upload=507714]: Upload finished. err: <nil> Duration: 30.00324277s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 3.584454477s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:39:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.584486648s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 3.39453502s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:38:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.394566089s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 35.825453372s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:38:58 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 35.825480924s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..X.X..X.XXX..X.....X.X.....X...XX.X......XXX.XX.X.XX.......] duration: 9.842411349s trace: https://jaeger.dev.storj.io/trace/6797d06f274d9c53 FAIL (2025-12-29 18:38:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: VHAVEDXGATIR6DBXKM4XKUQMFLEJJLWVI4C4S46ZGCG6ZPB3V5CA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: RAGXWVKLPDLNX2HNT34PUH6MZERHLCLSPMV75MQDF3MCCJEZQTIA) context deadline exceeded ======LOGS====== Dec 29 18:32:51.953251257 (579h3m51.824334302s) [upload=507625]: Starting upload Dec 29 18:32:51.953317824 (579h3m51.824400859s) [upload=507625]: Done starting upload Dec 29 18:32:51.953396002 (579h3m51.824479037s) [upload=507625]: Uploading segments... Dec 29 18:32:51.953668171 (579h3m51.824751216s) [upload=507625]: Got next segment. Inline: false Dec 29 18:32:51.953747421 (579h3m51.824830456s) [upload=507625]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:32:51.953752701 (579h3m51.824835746s) [upload=507625]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:32:52.202773254 (579h3m52.073856289s) [seg_pos={0 0}, upload=507625]: Begin upload segment... Dec 29 18:32:52.202827187 (579h3m52.073910222s) [seg_pos={0 0}, upload=507625]: Done begin upload segment. Dec 29 18:32:52.202859569 (579h3m52.073942604s) [noise=true , address=77.90.31.19:4842 , piece_id=2XCVPARINYJJP6Q5RL2ROS6YE2RJ4KR7WKCH2IY5XKH4VYLSP6NQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.202976842 (579h3m52.074059887s) [noise=true , address=77.90.31.14:4661 , piece_id=3QU3JVKHPZNTQPPRE6R5LPHO27JGARAESXVEIUXJ7CHHNEIGVKSA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.203661315 (579h3m52.07474435s) [noise=true , address=145.79.227.22:4510 , piece_id=WIAUNELPEZP4OF64F4Z6IEGPDZM6ODUODHQFHEGJOFIG7KYKAFZA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.203708254 (579h3m52.074791299s) [noise=true , address=145.79.227.24:4581 , piece_id=MPFIHZZSEVR6ESIP5QSH5GGIFGE6KMCJ2DAR5ZSS66P66KDPAHFA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.204243224 (579h3m52.075326269s) [noise=true , address=145.79.227.37:4565 , piece_id=KXKUL4642LGR5EFTFVVBOSYDTPDUCEZUNKUQS65KPQGDZDPVVV5Q, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.204825231 (579h3m52.075908276s) [noise=true , address=77.90.31.21:4880 , piece_id=BBZWEOHCQUZPZFYKH232RG6IFQEDA4TLNZ3HFRP2OHGMYIRMGSCA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.205446635 (579h3m52.07652967s) [noise=true , address=45.91.116.7:4009 , piece_id=VHAVEDXGATIR6DBXKM4XKUQMFLEJJLWVI4C4S46ZGCG6ZPB3V5CA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206012042 (579h3m52.077095077s) [noise=true , address=145.79.227.21:4475 , piece_id=JQD73PQDYUD4TZGIQBVSSZPNU5P7GESKX3GTA4PYB2HHIY4UB3KA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206118475 (579h3m52.07720151s) [noise=true , address=77.90.31.11:4555 , piece_id=VPILVCKGN2UK5RPLNNU3PQC72L7QN63ZXTG3435ZHMX45CWVH6DA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206425488 (579h3m52.077508523s) [noise=true , address=145.79.227.45:4229 , piece_id=C3CBOTVOIDTDCXJVNYUXQT5NREIOH5RA7EDD32LWS7WPIBHWYGMQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206745689 (579h3m52.077828734s) [noise=true , address=145.79.227.46:4602 , piece_id=6OJH335K6JBBPRIAL4QA5IWPSYN7SKEMPHR4ECWNSFDDVQQPD4TQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206758002 (579h3m52.077841037s) [noise=true , address=145.79.227.35:4483 , piece_id=ZJ7LEWE4ENTB2SHIXRPY3ASB6YHGJZ2BUJSJG4MYF53V6V4GFNGQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.207067291 (579h3m52.078150326s) [noise=true , address=45.91.116.9:4261 , piece_id=RAGXWVKLPDLNX2HNT34PUH6MZERHLCLSPMV75MQDF3MCCJEZQTIA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.207381760 (579h3m52.078464805s) [noise=true , address=145.79.227.44:4194 , piece_id=2Y43LHVPV53UAXRQOBBWTLAHR35AECGC3OE265TUMEGTD6MZYUEQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.207408040 (579h3m52.078491075s) [noise=true , address=145.79.227.5:4223 , piece_id=YVIFYEU2WJP6VP4YHQMOZV2SWPJLGLDJAWXMV352AC6WSDXMFVFA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.208987507 (579h3m52.080070542s) [noise=true , address=45.91.116.12:4227 , piece_id=CWF3YRREALI42IMYEPQZ6W3QWM6XFQKD2Z33DRUWVNEBLWS7OY7A, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:53.910451675 (579h3m53.78153472s) [upload=507625]: Next returned nil segment Dec 29 18:32:53.910474970 (579h3m53.781558005s) [upload=507625]: Waiting for error group managing segments... Dec 29 18:32:54.188945591 (579h3m54.060028626s) [noise=true , address=145.79.227.24:4581 , piece_id=MPFIHZZSEVR6ESIP5QSH5GGIFGE6KMCJ2DAR5ZSS66P66KDPAHFA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.196416346 (579h3m54.067499391s) [noise=true , address=145.79.227.45:4229 , piece_id=C3CBOTVOIDTDCXJVNYUXQT5NREIOH5RA7EDD32LWS7WPIBHWYGMQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.197993860 (579h3m54.069076895s) [noise=true , address=145.79.227.46:4602 , piece_id=6OJH335K6JBBPRIAL4QA5IWPSYN7SKEMPHR4ECWNSFDDVQQPD4TQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.200957113 (579h3m54.072040148s) [noise=true , address=145.79.227.22:4510 , piece_id=WIAUNELPEZP4OF64F4Z6IEGPDZM6ODUODHQFHEGJOFIG7KYKAFZA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.207922645 (579h3m54.07900569s) [noise=true , address=145.79.227.37:4565 , piece_id=KXKUL4642LGR5EFTFVVBOSYDTPDUCEZUNKUQS65KPQGDZDPVVV5Q, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.209264250 (579h3m54.080347285s) [noise=true , address=145.79.227.21:4475 , piece_id=JQD73PQDYUD4TZGIQBVSSZPNU5P7GESKX3GTA4PYB2HHIY4UB3KA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.211584358 (579h3m54.082667393s) [noise=true , address=145.79.227.5:4223 , piece_id=YVIFYEU2WJP6VP4YHQMOZV2SWPJLGLDJAWXMV352AC6WSDXMFVFA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.368210027 (579h3m54.239293062s) [noise=true , address=77.90.31.19:4842 , piece_id=2XCVPARINYJJP6Q5RL2ROS6YE2RJ4KR7WKCH2IY5XKH4VYLSP6NQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.431661819 (579h3m54.302744864s) [noise=true , address=77.90.31.11:4555 , piece_id=VPILVCKGN2UK5RPLNNU3PQC72L7QN63ZXTG3435ZHMX45CWVH6DA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.492327486 (579h3m54.363410521s) [noise=true , address=77.90.31.21:4880 , piece_id=BBZWEOHCQUZPZFYKH232RG6IFQEDA4TLNZ3HFRP2OHGMYIRMGSCA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.508530138 (579h3m54.379613173s) [noise=true , address=45.91.116.12:4227 , piece_id=CWF3YRREALI42IMYEPQZ6W3QWM6XFQKD2Z33DRUWVNEBLWS7OY7A, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.535358383 (579h3m54.406441418s) [noise=true , address=45.91.116.7:4009 , piece_id=VHAVEDXGATIR6DBXKM4XKUQMFLEJJLWVI4C4S46ZGCG6ZPB3V5CA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.579972706 (579h3m54.451055741s) [noise=true , address=45.91.116.9:4261 , piece_id=RAGXWVKLPDLNX2HNT34PUH6MZERHLCLSPMV75MQDF3MCCJEZQTIA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.580000098 (579h3m54.451083143s) [seg_pos={0 0}, upload=507625]: Segment reached optimal threshold of 13 pieces. Dec 29 18:32:54.580238622 (579h3m54.451321657s) [noise=true , address=145.79.227.44:4194 , piece_id=2Y43LHVPV53UAXRQOBBWTLAHR35AECGC3OE265TUMEGTD6MZYUEQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 18:32:54.580248180 (579h3m54.451331216s) [noise=true , address=145.79.227.44:4194 , piece_id=2Y43LHVPV53UAXRQOBBWTLAHR35AECGC3OE265TUMEGTD6MZYUEQ, seg_pos={0 0}, upload=507625]: Long tail cancellation due to optimal threshold Dec 29 18:32:54.580453021 (579h3m54.451536056s) [noise=true , address=77.90.31.14:4661 , piece_id=3QU3JVKHPZNTQPPRE6R5LPHO27JGARAESXVEIUXJ7CHHNEIGVKSA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 29 18:32:54.580459012 (579h3m54.451542047s) [noise=true , address=77.90.31.14:4661 , piece_id=3QU3JVKHPZNTQPPRE6R5LPHO27JGARAESXVEIUXJ7CHHNEIGVKSA, seg_pos={0 0}, upload=507625]: Long tail cancellation due to optimal threshold Dec 29 18:32:54.580481203 (579h3m54.451564259s) [noise=true , address=145.79.227.35:4483 , piece_id=ZJ7LEWE4ENTB2SHIXRPY3ASB6YHGJZ2BUJSJG4MYF53V6V4GFNGQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 29 18:32:54.580500681 (579h3m54.451583716s) [noise=true , address=145.79.227.35:4483 , piece_id=ZJ7LEWE4ENTB2SHIXRPY3ASB6YHGJZ2BUJSJG4MYF53V6V4GFNGQ, seg_pos={0 0}, upload=507625]: Long tail cancellation due to optimal threshold Dec 29 18:32:54.580533243 (579h3m54.451616288s) [seg_pos={0 0}, upload=507625]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:32:54.580611903 (579h3m54.451694938s) [upload=507625]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:32:54.580618224 (579h3m54.45170127s) [upload=507625]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:32:54.878900119 (579h3m54.749983154s) [upload=507625]: Done uploading segments... Dec 29 18:32:54.878920508 (579h3m54.750003553s) [upload=507625]: Upload finished. err: <nil> Duration: 9.842431256s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [...X.X..X.XXX..X.....X.X.....X...XX.X......XXX.XX.X.XX......] duration: 6.025738586s trace: https://jaeger.dev.storj.io/trace/6797d06f274d9c53 FAIL (2025-12-29 18:38:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: VHAVEDXGATIR6DBXKM4XKUQMFLEJJLWVI4C4S46ZGCG6ZPB3V5CA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: RAGXWVKLPDLNX2HNT34PUH6MZERHLCLSPMV75MQDF3MCCJEZQTIA) context deadline exceeded ======LOGS====== Dec 29 18:32:51.953251257 (579h3m51.824334302s) [upload=507625]: Starting upload Dec 29 18:32:51.953317824 (579h3m51.824400859s) [upload=507625]: Done starting upload Dec 29 18:32:51.953396002 (579h3m51.824479037s) [upload=507625]: Uploading segments... Dec 29 18:32:51.953668171 (579h3m51.824751216s) [upload=507625]: Got next segment. Inline: false Dec 29 18:32:51.953747421 (579h3m51.824830456s) [upload=507625]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:32:51.953752701 (579h3m51.824835746s) [upload=507625]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:32:52.202773254 (579h3m52.073856289s) [seg_pos={0 0}, upload=507625]: Begin upload segment... Dec 29 18:32:52.202827187 (579h3m52.073910222s) [seg_pos={0 0}, upload=507625]: Done begin upload segment. Dec 29 18:32:52.202859569 (579h3m52.073942604s) [noise=true , address=77.90.31.19:4842 , piece_id=2XCVPARINYJJP6Q5RL2ROS6YE2RJ4KR7WKCH2IY5XKH4VYLSP6NQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.202976842 (579h3m52.074059887s) [noise=true , address=77.90.31.14:4661 , piece_id=3QU3JVKHPZNTQPPRE6R5LPHO27JGARAESXVEIUXJ7CHHNEIGVKSA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.203661315 (579h3m52.07474435s) [noise=true , address=145.79.227.22:4510 , piece_id=WIAUNELPEZP4OF64F4Z6IEGPDZM6ODUODHQFHEGJOFIG7KYKAFZA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.203708254 (579h3m52.074791299s) [noise=true , address=145.79.227.24:4581 , piece_id=MPFIHZZSEVR6ESIP5QSH5GGIFGE6KMCJ2DAR5ZSS66P66KDPAHFA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.204243224 (579h3m52.075326269s) [noise=true , address=145.79.227.37:4565 , piece_id=KXKUL4642LGR5EFTFVVBOSYDTPDUCEZUNKUQS65KPQGDZDPVVV5Q, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.204825231 (579h3m52.075908276s) [noise=true , address=77.90.31.21:4880 , piece_id=BBZWEOHCQUZPZFYKH232RG6IFQEDA4TLNZ3HFRP2OHGMYIRMGSCA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.205446635 (579h3m52.07652967s) [noise=true , address=45.91.116.7:4009 , piece_id=VHAVEDXGATIR6DBXKM4XKUQMFLEJJLWVI4C4S46ZGCG6ZPB3V5CA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206012042 (579h3m52.077095077s) [noise=true , address=145.79.227.21:4475 , piece_id=JQD73PQDYUD4TZGIQBVSSZPNU5P7GESKX3GTA4PYB2HHIY4UB3KA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206118475 (579h3m52.07720151s) [noise=true , address=77.90.31.11:4555 , piece_id=VPILVCKGN2UK5RPLNNU3PQC72L7QN63ZXTG3435ZHMX45CWVH6DA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206425488 (579h3m52.077508523s) [noise=true , address=145.79.227.45:4229 , piece_id=C3CBOTVOIDTDCXJVNYUXQT5NREIOH5RA7EDD32LWS7WPIBHWYGMQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206745689 (579h3m52.077828734s) [noise=true , address=145.79.227.46:4602 , piece_id=6OJH335K6JBBPRIAL4QA5IWPSYN7SKEMPHR4ECWNSFDDVQQPD4TQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206758002 (579h3m52.077841037s) [noise=true , address=145.79.227.35:4483 , piece_id=ZJ7LEWE4ENTB2SHIXRPY3ASB6YHGJZ2BUJSJG4MYF53V6V4GFNGQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.207067291 (579h3m52.078150326s) [noise=true , address=45.91.116.9:4261 , piece_id=RAGXWVKLPDLNX2HNT34PUH6MZERHLCLSPMV75MQDF3MCCJEZQTIA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.207381760 (579h3m52.078464805s) [noise=true , address=145.79.227.44:4194 , piece_id=2Y43LHVPV53UAXRQOBBWTLAHR35AECGC3OE265TUMEGTD6MZYUEQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.207408040 (579h3m52.078491075s) [noise=true , address=145.79.227.5:4223 , piece_id=YVIFYEU2WJP6VP4YHQMOZV2SWPJLGLDJAWXMV352AC6WSDXMFVFA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.208987507 (579h3m52.080070542s) [noise=true , address=45.91.116.12:4227 , piece_id=CWF3YRREALI42IMYEPQZ6W3QWM6XFQKD2Z33DRUWVNEBLWS7OY7A, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:53.910451675 (579h3m53.78153472s) [upload=507625]: Next returned nil segment Dec 29 18:32:53.910474970 (579h3m53.781558005s) [upload=507625]: Waiting for error group managing segments... Dec 29 18:32:54.188945591 (579h3m54.060028626s) [noise=true , address=145.79.227.24:4581 , piece_id=MPFIHZZSEVR6ESIP5QSH5GGIFGE6KMCJ2DAR5ZSS66P66KDPAHFA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.196416346 (579h3m54.067499391s) [noise=true , address=145.79.227.45:4229 , piece_id=C3CBOTVOIDTDCXJVNYUXQT5NREIOH5RA7EDD32LWS7WPIBHWYGMQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.197993860 (579h3m54.069076895s) [noise=true , address=145.79.227.46:4602 , piece_id=6OJH335K6JBBPRIAL4QA5IWPSYN7SKEMPHR4ECWNSFDDVQQPD4TQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.200957113 (579h3m54.072040148s) [noise=true , address=145.79.227.22:4510 , piece_id=WIAUNELPEZP4OF64F4Z6IEGPDZM6ODUODHQFHEGJOFIG7KYKAFZA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.207922645 (579h3m54.07900569s) [noise=true , address=145.79.227.37:4565 , piece_id=KXKUL4642LGR5EFTFVVBOSYDTPDUCEZUNKUQS65KPQGDZDPVVV5Q, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.209264250 (579h3m54.080347285s) [noise=true , address=145.79.227.21:4475 , piece_id=JQD73PQDYUD4TZGIQBVSSZPNU5P7GESKX3GTA4PYB2HHIY4UB3KA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.211584358 (579h3m54.082667393s) [noise=true , address=145.79.227.5:4223 , piece_id=YVIFYEU2WJP6VP4YHQMOZV2SWPJLGLDJAWXMV352AC6WSDXMFVFA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.368210027 (579h3m54.239293062s) [noise=true , address=77.90.31.19:4842 , piece_id=2XCVPARINYJJP6Q5RL2ROS6YE2RJ4KR7WKCH2IY5XKH4VYLSP6NQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.431661819 (579h3m54.302744864s) [noise=true , address=77.90.31.11:4555 , piece_id=VPILVCKGN2UK5RPLNNU3PQC72L7QN63ZXTG3435ZHMX45CWVH6DA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.492327486 (579h3m54.363410521s) [noise=true , address=77.90.31.21:4880 , piece_id=BBZWEOHCQUZPZFYKH232RG6IFQEDA4TLNZ3HFRP2OHGMYIRMGSCA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.508530138 (579h3m54.379613173s) [noise=true , address=45.91.116.12:4227 , piece_id=CWF3YRREALI42IMYEPQZ6W3QWM6XFQKD2Z33DRUWVNEBLWS7OY7A, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.535358383 (579h3m54.406441418s) [noise=true , address=45.91.116.7:4009 , piece_id=VHAVEDXGATIR6DBXKM4XKUQMFLEJJLWVI4C4S46ZGCG6ZPB3V5CA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.579972706 (579h3m54.451055741s) [noise=true , address=45.91.116.9:4261 , piece_id=RAGXWVKLPDLNX2HNT34PUH6MZERHLCLSPMV75MQDF3MCCJEZQTIA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.580000098 (579h3m54.451083143s) [seg_pos={0 0}, upload=507625]: Segment reached optimal threshold of 13 pieces. Dec 29 18:32:54.580238622 (579h3m54.451321657s) [noise=true , address=145.79.227.44:4194 , piece_id=2Y43LHVPV53UAXRQOBBWTLAHR35AECGC3OE265TUMEGTD6MZYUEQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 18:32:54.580248180 (579h3m54.451331216s) [noise=true , address=145.79.227.44:4194 , piece_id=2Y43LHVPV53UAXRQOBBWTLAHR35AECGC3OE265TUMEGTD6MZYUEQ, seg_pos={0 0}, upload=507625]: Long tail cancellation due to optimal threshold Dec 29 18:32:54.580453021 (579h3m54.451536056s) [noise=true , address=77.90.31.14:4661 , piece_id=3QU3JVKHPZNTQPPRE6R5LPHO27JGARAESXVEIUXJ7CHHNEIGVKSA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 29 18:32:54.580459012 (579h3m54.451542047s) [noise=true , address=77.90.31.14:4661 , piece_id=3QU3JVKHPZNTQPPRE6R5LPHO27JGARAESXVEIUXJ7CHHNEIGVKSA, seg_pos={0 0}, upload=507625]: Long tail cancellation due to optimal threshold Dec 29 18:32:54.580481203 (579h3m54.451564259s) [noise=true , address=145.79.227.35:4483 , piece_id=ZJ7LEWE4ENTB2SHIXRPY3ASB6YHGJZ2BUJSJG4MYF53V6V4GFNGQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 29 18:32:54.580500681 (579h3m54.451583716s) [noise=true , address=145.79.227.35:4483 , piece_id=ZJ7LEWE4ENTB2SHIXRPY3ASB6YHGJZ2BUJSJG4MYF53V6V4GFNGQ, seg_pos={0 0}, upload=507625]: Long tail cancellation due to optimal threshold Dec 29 18:32:54.580533243 (579h3m54.451616288s) [seg_pos={0 0}, upload=507625]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:32:54.580611903 (579h3m54.451694938s) [upload=507625]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:32:54.580618224 (579h3m54.45170127s) [upload=507625]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:32:54.878900119 (579h3m54.749983154s) [upload=507625]: Done uploading segments... Dec 29 18:32:54.878920508 (579h3m54.750003553s) [upload=507625]: Upload finished. err: <nil> Duration: 6.025797208s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 3.641385802s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:37:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.641412023s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 25.044123909s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:37:44 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 25.044138617s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 30.290809867s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:37:19 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 30.290836047s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [....X.X..X.XXX..X.....X.X.....X...XX.X......XXX.XX.X.XX.....] duration: 6.048956866s trace: https://jaeger.dev.storj.io/trace/6797d06f274d9c53 FAIL (2025-12-29 18:37:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: VHAVEDXGATIR6DBXKM4XKUQMFLEJJLWVI4C4S46ZGCG6ZPB3V5CA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: RAGXWVKLPDLNX2HNT34PUH6MZERHLCLSPMV75MQDF3MCCJEZQTIA) context deadline exceeded ======LOGS====== Dec 29 18:32:51.953251257 (579h3m51.824334302s) [upload=507625]: Starting upload Dec 29 18:32:51.953317824 (579h3m51.824400859s) [upload=507625]: Done starting upload Dec 29 18:32:51.953396002 (579h3m51.824479037s) [upload=507625]: Uploading segments... Dec 29 18:32:51.953668171 (579h3m51.824751216s) [upload=507625]: Got next segment. Inline: false Dec 29 18:32:51.953747421 (579h3m51.824830456s) [upload=507625]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:32:51.953752701 (579h3m51.824835746s) [upload=507625]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:32:52.202773254 (579h3m52.073856289s) [seg_pos={0 0}, upload=507625]: Begin upload segment... Dec 29 18:32:52.202827187 (579h3m52.073910222s) [seg_pos={0 0}, upload=507625]: Done begin upload segment. Dec 29 18:32:52.202859569 (579h3m52.073942604s) [noise=true , address=77.90.31.19:4842 , piece_id=2XCVPARINYJJP6Q5RL2ROS6YE2RJ4KR7WKCH2IY5XKH4VYLSP6NQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.202976842 (579h3m52.074059887s) [noise=true , address=77.90.31.14:4661 , piece_id=3QU3JVKHPZNTQPPRE6R5LPHO27JGARAESXVEIUXJ7CHHNEIGVKSA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.203661315 (579h3m52.07474435s) [noise=true , address=145.79.227.22:4510 , piece_id=WIAUNELPEZP4OF64F4Z6IEGPDZM6ODUODHQFHEGJOFIG7KYKAFZA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.203708254 (579h3m52.074791299s) [noise=true , address=145.79.227.24:4581 , piece_id=MPFIHZZSEVR6ESIP5QSH5GGIFGE6KMCJ2DAR5ZSS66P66KDPAHFA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.204243224 (579h3m52.075326269s) [noise=true , address=145.79.227.37:4565 , piece_id=KXKUL4642LGR5EFTFVVBOSYDTPDUCEZUNKUQS65KPQGDZDPVVV5Q, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.204825231 (579h3m52.075908276s) [noise=true , address=77.90.31.21:4880 , piece_id=BBZWEOHCQUZPZFYKH232RG6IFQEDA4TLNZ3HFRP2OHGMYIRMGSCA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.205446635 (579h3m52.07652967s) [noise=true , address=45.91.116.7:4009 , piece_id=VHAVEDXGATIR6DBXKM4XKUQMFLEJJLWVI4C4S46ZGCG6ZPB3V5CA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206012042 (579h3m52.077095077s) [noise=true , address=145.79.227.21:4475 , piece_id=JQD73PQDYUD4TZGIQBVSSZPNU5P7GESKX3GTA4PYB2HHIY4UB3KA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206118475 (579h3m52.07720151s) [noise=true , address=77.90.31.11:4555 , piece_id=VPILVCKGN2UK5RPLNNU3PQC72L7QN63ZXTG3435ZHMX45CWVH6DA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206425488 (579h3m52.077508523s) [noise=true , address=145.79.227.45:4229 , piece_id=C3CBOTVOIDTDCXJVNYUXQT5NREIOH5RA7EDD32LWS7WPIBHWYGMQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206745689 (579h3m52.077828734s) [noise=true , address=145.79.227.46:4602 , piece_id=6OJH335K6JBBPRIAL4QA5IWPSYN7SKEMPHR4ECWNSFDDVQQPD4TQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206758002 (579h3m52.077841037s) [noise=true , address=145.79.227.35:4483 , piece_id=ZJ7LEWE4ENTB2SHIXRPY3ASB6YHGJZ2BUJSJG4MYF53V6V4GFNGQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.207067291 (579h3m52.078150326s) [noise=true , address=45.91.116.9:4261 , piece_id=RAGXWVKLPDLNX2HNT34PUH6MZERHLCLSPMV75MQDF3MCCJEZQTIA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.207381760 (579h3m52.078464805s) [noise=true , address=145.79.227.44:4194 , piece_id=2Y43LHVPV53UAXRQOBBWTLAHR35AECGC3OE265TUMEGTD6MZYUEQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.207408040 (579h3m52.078491075s) [noise=true , address=145.79.227.5:4223 , piece_id=YVIFYEU2WJP6VP4YHQMOZV2SWPJLGLDJAWXMV352AC6WSDXMFVFA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.208987507 (579h3m52.080070542s) [noise=true , address=45.91.116.12:4227 , piece_id=CWF3YRREALI42IMYEPQZ6W3QWM6XFQKD2Z33DRUWVNEBLWS7OY7A, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:53.910451675 (579h3m53.78153472s) [upload=507625]: Next returned nil segment Dec 29 18:32:53.910474970 (579h3m53.781558005s) [upload=507625]: Waiting for error group managing segments... Dec 29 18:32:54.188945591 (579h3m54.060028626s) [noise=true , address=145.79.227.24:4581 , piece_id=MPFIHZZSEVR6ESIP5QSH5GGIFGE6KMCJ2DAR5ZSS66P66KDPAHFA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.196416346 (579h3m54.067499391s) [noise=true , address=145.79.227.45:4229 , piece_id=C3CBOTVOIDTDCXJVNYUXQT5NREIOH5RA7EDD32LWS7WPIBHWYGMQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.197993860 (579h3m54.069076895s) [noise=true , address=145.79.227.46:4602 , piece_id=6OJH335K6JBBPRIAL4QA5IWPSYN7SKEMPHR4ECWNSFDDVQQPD4TQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.200957113 (579h3m54.072040148s) [noise=true , address=145.79.227.22:4510 , piece_id=WIAUNELPEZP4OF64F4Z6IEGPDZM6ODUODHQFHEGJOFIG7KYKAFZA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.207922645 (579h3m54.07900569s) [noise=true , address=145.79.227.37:4565 , piece_id=KXKUL4642LGR5EFTFVVBOSYDTPDUCEZUNKUQS65KPQGDZDPVVV5Q, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.209264250 (579h3m54.080347285s) [noise=true , address=145.79.227.21:4475 , piece_id=JQD73PQDYUD4TZGIQBVSSZPNU5P7GESKX3GTA4PYB2HHIY4UB3KA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.211584358 (579h3m54.082667393s) [noise=true , address=145.79.227.5:4223 , piece_id=YVIFYEU2WJP6VP4YHQMOZV2SWPJLGLDJAWXMV352AC6WSDXMFVFA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.368210027 (579h3m54.239293062s) [noise=true , address=77.90.31.19:4842 , piece_id=2XCVPARINYJJP6Q5RL2ROS6YE2RJ4KR7WKCH2IY5XKH4VYLSP6NQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.431661819 (579h3m54.302744864s) [noise=true , address=77.90.31.11:4555 , piece_id=VPILVCKGN2UK5RPLNNU3PQC72L7QN63ZXTG3435ZHMX45CWVH6DA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.492327486 (579h3m54.363410521s) [noise=true , address=77.90.31.21:4880 , piece_id=BBZWEOHCQUZPZFYKH232RG6IFQEDA4TLNZ3HFRP2OHGMYIRMGSCA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.508530138 (579h3m54.379613173s) [noise=true , address=45.91.116.12:4227 , piece_id=CWF3YRREALI42IMYEPQZ6W3QWM6XFQKD2Z33DRUWVNEBLWS7OY7A, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.535358383 (579h3m54.406441418s) [noise=true , address=45.91.116.7:4009 , piece_id=VHAVEDXGATIR6DBXKM4XKUQMFLEJJLWVI4C4S46ZGCG6ZPB3V5CA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.579972706 (579h3m54.451055741s) [noise=true , address=45.91.116.9:4261 , piece_id=RAGXWVKLPDLNX2HNT34PUH6MZERHLCLSPMV75MQDF3MCCJEZQTIA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.580000098 (579h3m54.451083143s) [seg_pos={0 0}, upload=507625]: Segment reached optimal threshold of 13 pieces. Dec 29 18:32:54.580238622 (579h3m54.451321657s) [noise=true , address=145.79.227.44:4194 , piece_id=2Y43LHVPV53UAXRQOBBWTLAHR35AECGC3OE265TUMEGTD6MZYUEQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 18:32:54.580248180 (579h3m54.451331216s) [noise=true , address=145.79.227.44:4194 , piece_id=2Y43LHVPV53UAXRQOBBWTLAHR35AECGC3OE265TUMEGTD6MZYUEQ, seg_pos={0 0}, upload=507625]: Long tail cancellation due to optimal threshold Dec 29 18:32:54.580453021 (579h3m54.451536056s) [noise=true , address=77.90.31.14:4661 , piece_id=3QU3JVKHPZNTQPPRE6R5LPHO27JGARAESXVEIUXJ7CHHNEIGVKSA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 29 18:32:54.580459012 (579h3m54.451542047s) [noise=true , address=77.90.31.14:4661 , piece_id=3QU3JVKHPZNTQPPRE6R5LPHO27JGARAESXVEIUXJ7CHHNEIGVKSA, seg_pos={0 0}, upload=507625]: Long tail cancellation due to optimal threshold Dec 29 18:32:54.580481203 (579h3m54.451564259s) [noise=true , address=145.79.227.35:4483 , piece_id=ZJ7LEWE4ENTB2SHIXRPY3ASB6YHGJZ2BUJSJG4MYF53V6V4GFNGQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 29 18:32:54.580500681 (579h3m54.451583716s) [noise=true , address=145.79.227.35:4483 , piece_id=ZJ7LEWE4ENTB2SHIXRPY3ASB6YHGJZ2BUJSJG4MYF53V6V4GFNGQ, seg_pos={0 0}, upload=507625]: Long tail cancellation due to optimal threshold Dec 29 18:32:54.580533243 (579h3m54.451616288s) [seg_pos={0 0}, upload=507625]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:32:54.580611903 (579h3m54.451694938s) [upload=507625]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:32:54.580618224 (579h3m54.45170127s) [upload=507625]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:32:54.878900119 (579h3m54.749983154s) [upload=507625]: Done uploading segments... Dec 29 18:32:54.878920508 (579h3m54.750003553s) [upload=507625]: Upload finished. err: <nil> Duration: 6.048997374s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 3.549750705s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:36:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.549777536s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 22.181070058s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:36:28 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 22.181089756s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X....X.X..X.XXX..X.....X.X.....X...XX.X......XXX.XX.X.XX....] duration: 6.062221592s trace: https://jaeger.dev.storj.io/trace/6797d06f274d9c53 FAIL (2025-12-29 18:36:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: VHAVEDXGATIR6DBXKM4XKUQMFLEJJLWVI4C4S46ZGCG6ZPB3V5CA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: RAGXWVKLPDLNX2HNT34PUH6MZERHLCLSPMV75MQDF3MCCJEZQTIA) context deadline exceeded ======LOGS====== Dec 29 18:32:51.953251257 (579h3m51.824334302s) [upload=507625]: Starting upload Dec 29 18:32:51.953317824 (579h3m51.824400859s) [upload=507625]: Done starting upload Dec 29 18:32:51.953396002 (579h3m51.824479037s) [upload=507625]: Uploading segments... Dec 29 18:32:51.953668171 (579h3m51.824751216s) [upload=507625]: Got next segment. Inline: false Dec 29 18:32:51.953747421 (579h3m51.824830456s) [upload=507625]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:32:51.953752701 (579h3m51.824835746s) [upload=507625]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:32:52.202773254 (579h3m52.073856289s) [seg_pos={0 0}, upload=507625]: Begin upload segment... Dec 29 18:32:52.202827187 (579h3m52.073910222s) [seg_pos={0 0}, upload=507625]: Done begin upload segment. Dec 29 18:32:52.202859569 (579h3m52.073942604s) [noise=true , address=77.90.31.19:4842 , piece_id=2XCVPARINYJJP6Q5RL2ROS6YE2RJ4KR7WKCH2IY5XKH4VYLSP6NQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.202976842 (579h3m52.074059887s) [noise=true , address=77.90.31.14:4661 , piece_id=3QU3JVKHPZNTQPPRE6R5LPHO27JGARAESXVEIUXJ7CHHNEIGVKSA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.203661315 (579h3m52.07474435s) [noise=true , address=145.79.227.22:4510 , piece_id=WIAUNELPEZP4OF64F4Z6IEGPDZM6ODUODHQFHEGJOFIG7KYKAFZA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.203708254 (579h3m52.074791299s) [noise=true , address=145.79.227.24:4581 , piece_id=MPFIHZZSEVR6ESIP5QSH5GGIFGE6KMCJ2DAR5ZSS66P66KDPAHFA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.204243224 (579h3m52.075326269s) [noise=true , address=145.79.227.37:4565 , piece_id=KXKUL4642LGR5EFTFVVBOSYDTPDUCEZUNKUQS65KPQGDZDPVVV5Q, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.204825231 (579h3m52.075908276s) [noise=true , address=77.90.31.21:4880 , piece_id=BBZWEOHCQUZPZFYKH232RG6IFQEDA4TLNZ3HFRP2OHGMYIRMGSCA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.205446635 (579h3m52.07652967s) [noise=true , address=45.91.116.7:4009 , piece_id=VHAVEDXGATIR6DBXKM4XKUQMFLEJJLWVI4C4S46ZGCG6ZPB3V5CA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206012042 (579h3m52.077095077s) [noise=true , address=145.79.227.21:4475 , piece_id=JQD73PQDYUD4TZGIQBVSSZPNU5P7GESKX3GTA4PYB2HHIY4UB3KA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206118475 (579h3m52.07720151s) [noise=true , address=77.90.31.11:4555 , piece_id=VPILVCKGN2UK5RPLNNU3PQC72L7QN63ZXTG3435ZHMX45CWVH6DA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206425488 (579h3m52.077508523s) [noise=true , address=145.79.227.45:4229 , piece_id=C3CBOTVOIDTDCXJVNYUXQT5NREIOH5RA7EDD32LWS7WPIBHWYGMQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206745689 (579h3m52.077828734s) [noise=true , address=145.79.227.46:4602 , piece_id=6OJH335K6JBBPRIAL4QA5IWPSYN7SKEMPHR4ECWNSFDDVQQPD4TQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206758002 (579h3m52.077841037s) [noise=true , address=145.79.227.35:4483 , piece_id=ZJ7LEWE4ENTB2SHIXRPY3ASB6YHGJZ2BUJSJG4MYF53V6V4GFNGQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.207067291 (579h3m52.078150326s) [noise=true , address=45.91.116.9:4261 , piece_id=RAGXWVKLPDLNX2HNT34PUH6MZERHLCLSPMV75MQDF3MCCJEZQTIA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.207381760 (579h3m52.078464805s) [noise=true , address=145.79.227.44:4194 , piece_id=2Y43LHVPV53UAXRQOBBWTLAHR35AECGC3OE265TUMEGTD6MZYUEQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.207408040 (579h3m52.078491075s) [noise=true , address=145.79.227.5:4223 , piece_id=YVIFYEU2WJP6VP4YHQMOZV2SWPJLGLDJAWXMV352AC6WSDXMFVFA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.208987507 (579h3m52.080070542s) [noise=true , address=45.91.116.12:4227 , piece_id=CWF3YRREALI42IMYEPQZ6W3QWM6XFQKD2Z33DRUWVNEBLWS7OY7A, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:53.910451675 (579h3m53.78153472s) [upload=507625]: Next returned nil segment Dec 29 18:32:53.910474970 (579h3m53.781558005s) [upload=507625]: Waiting for error group managing segments... Dec 29 18:32:54.188945591 (579h3m54.060028626s) [noise=true , address=145.79.227.24:4581 , piece_id=MPFIHZZSEVR6ESIP5QSH5GGIFGE6KMCJ2DAR5ZSS66P66KDPAHFA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.196416346 (579h3m54.067499391s) [noise=true , address=145.79.227.45:4229 , piece_id=C3CBOTVOIDTDCXJVNYUXQT5NREIOH5RA7EDD32LWS7WPIBHWYGMQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.197993860 (579h3m54.069076895s) [noise=true , address=145.79.227.46:4602 , piece_id=6OJH335K6JBBPRIAL4QA5IWPSYN7SKEMPHR4ECWNSFDDVQQPD4TQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.200957113 (579h3m54.072040148s) [noise=true , address=145.79.227.22:4510 , piece_id=WIAUNELPEZP4OF64F4Z6IEGPDZM6ODUODHQFHEGJOFIG7KYKAFZA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.207922645 (579h3m54.07900569s) [noise=true , address=145.79.227.37:4565 , piece_id=KXKUL4642LGR5EFTFVVBOSYDTPDUCEZUNKUQS65KPQGDZDPVVV5Q, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.209264250 (579h3m54.080347285s) [noise=true , address=145.79.227.21:4475 , piece_id=JQD73PQDYUD4TZGIQBVSSZPNU5P7GESKX3GTA4PYB2HHIY4UB3KA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.211584358 (579h3m54.082667393s) [noise=true , address=145.79.227.5:4223 , piece_id=YVIFYEU2WJP6VP4YHQMOZV2SWPJLGLDJAWXMV352AC6WSDXMFVFA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.368210027 (579h3m54.239293062s) [noise=true , address=77.90.31.19:4842 , piece_id=2XCVPARINYJJP6Q5RL2ROS6YE2RJ4KR7WKCH2IY5XKH4VYLSP6NQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.431661819 (579h3m54.302744864s) [noise=true , address=77.90.31.11:4555 , piece_id=VPILVCKGN2UK5RPLNNU3PQC72L7QN63ZXTG3435ZHMX45CWVH6DA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.492327486 (579h3m54.363410521s) [noise=true , address=77.90.31.21:4880 , piece_id=BBZWEOHCQUZPZFYKH232RG6IFQEDA4TLNZ3HFRP2OHGMYIRMGSCA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.508530138 (579h3m54.379613173s) [noise=true , address=45.91.116.12:4227 , piece_id=CWF3YRREALI42IMYEPQZ6W3QWM6XFQKD2Z33DRUWVNEBLWS7OY7A, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.535358383 (579h3m54.406441418s) [noise=true , address=45.91.116.7:4009 , piece_id=VHAVEDXGATIR6DBXKM4XKUQMFLEJJLWVI4C4S46ZGCG6ZPB3V5CA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.579972706 (579h3m54.451055741s) [noise=true , address=45.91.116.9:4261 , piece_id=RAGXWVKLPDLNX2HNT34PUH6MZERHLCLSPMV75MQDF3MCCJEZQTIA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.580000098 (579h3m54.451083143s) [seg_pos={0 0}, upload=507625]: Segment reached optimal threshold of 13 pieces. Dec 29 18:32:54.580238622 (579h3m54.451321657s) [noise=true , address=145.79.227.44:4194 , piece_id=2Y43LHVPV53UAXRQOBBWTLAHR35AECGC3OE265TUMEGTD6MZYUEQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 18:32:54.580248180 (579h3m54.451331216s) [noise=true , address=145.79.227.44:4194 , piece_id=2Y43LHVPV53UAXRQOBBWTLAHR35AECGC3OE265TUMEGTD6MZYUEQ, seg_pos={0 0}, upload=507625]: Long tail cancellation due to optimal threshold Dec 29 18:32:54.580453021 (579h3m54.451536056s) [noise=true , address=77.90.31.14:4661 , piece_id=3QU3JVKHPZNTQPPRE6R5LPHO27JGARAESXVEIUXJ7CHHNEIGVKSA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 29 18:32:54.580459012 (579h3m54.451542047s) [noise=true , address=77.90.31.14:4661 , piece_id=3QU3JVKHPZNTQPPRE6R5LPHO27JGARAESXVEIUXJ7CHHNEIGVKSA, seg_pos={0 0}, upload=507625]: Long tail cancellation due to optimal threshold Dec 29 18:32:54.580481203 (579h3m54.451564259s) [noise=true , address=145.79.227.35:4483 , piece_id=ZJ7LEWE4ENTB2SHIXRPY3ASB6YHGJZ2BUJSJG4MYF53V6V4GFNGQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 29 18:32:54.580500681 (579h3m54.451583716s) [noise=true , address=145.79.227.35:4483 , piece_id=ZJ7LEWE4ENTB2SHIXRPY3ASB6YHGJZ2BUJSJG4MYF53V6V4GFNGQ, seg_pos={0 0}, upload=507625]: Long tail cancellation due to optimal threshold Dec 29 18:32:54.580533243 (579h3m54.451616288s) [seg_pos={0 0}, upload=507625]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:32:54.580611903 (579h3m54.451694938s) [upload=507625]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:32:54.580618224 (579h3m54.45170127s) [upload=507625]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:32:54.878900119 (579h3m54.749983154s) [upload=507625]: Done uploading segments... Dec 29 18:32:54.878920508 (579h3m54.750003553s) [upload=507625]: Upload finished. err: <nil> Duration: 6.062250307s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 3.490412047s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:36:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.490441092s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X....X.X..X.XXX..X.....X.X.....X...XX.X......XXX.XX.X.XX...] duration: 12.896177369s trace: https://jaeger.dev.storj.io/trace/6797d06f274d9c53 FAIL (2025-12-29 18:35:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: VHAVEDXGATIR6DBXKM4XKUQMFLEJJLWVI4C4S46ZGCG6ZPB3V5CA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: RAGXWVKLPDLNX2HNT34PUH6MZERHLCLSPMV75MQDF3MCCJEZQTIA) context deadline exceeded ======LOGS====== Dec 29 18:32:51.953251257 (579h3m51.824334302s) [upload=507625]: Starting upload Dec 29 18:32:51.953317824 (579h3m51.824400859s) [upload=507625]: Done starting upload Dec 29 18:32:51.953396002 (579h3m51.824479037s) [upload=507625]: Uploading segments... Dec 29 18:32:51.953668171 (579h3m51.824751216s) [upload=507625]: Got next segment. Inline: false Dec 29 18:32:51.953747421 (579h3m51.824830456s) [upload=507625]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:32:51.953752701 (579h3m51.824835746s) [upload=507625]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:32:52.202773254 (579h3m52.073856289s) [seg_pos={0 0}, upload=507625]: Begin upload segment... Dec 29 18:32:52.202827187 (579h3m52.073910222s) [seg_pos={0 0}, upload=507625]: Done begin upload segment. Dec 29 18:32:52.202859569 (579h3m52.073942604s) [noise=true , address=77.90.31.19:4842 , piece_id=2XCVPARINYJJP6Q5RL2ROS6YE2RJ4KR7WKCH2IY5XKH4VYLSP6NQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.202976842 (579h3m52.074059887s) [noise=true , address=77.90.31.14:4661 , piece_id=3QU3JVKHPZNTQPPRE6R5LPHO27JGARAESXVEIUXJ7CHHNEIGVKSA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.203661315 (579h3m52.07474435s) [noise=true , address=145.79.227.22:4510 , piece_id=WIAUNELPEZP4OF64F4Z6IEGPDZM6ODUODHQFHEGJOFIG7KYKAFZA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.203708254 (579h3m52.074791299s) [noise=true , address=145.79.227.24:4581 , piece_id=MPFIHZZSEVR6ESIP5QSH5GGIFGE6KMCJ2DAR5ZSS66P66KDPAHFA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.204243224 (579h3m52.075326269s) [noise=true , address=145.79.227.37:4565 , piece_id=KXKUL4642LGR5EFTFVVBOSYDTPDUCEZUNKUQS65KPQGDZDPVVV5Q, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.204825231 (579h3m52.075908276s) [noise=true , address=77.90.31.21:4880 , piece_id=BBZWEOHCQUZPZFYKH232RG6IFQEDA4TLNZ3HFRP2OHGMYIRMGSCA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.205446635 (579h3m52.07652967s) [noise=true , address=45.91.116.7:4009 , piece_id=VHAVEDXGATIR6DBXKM4XKUQMFLEJJLWVI4C4S46ZGCG6ZPB3V5CA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206012042 (579h3m52.077095077s) [noise=true , address=145.79.227.21:4475 , piece_id=JQD73PQDYUD4TZGIQBVSSZPNU5P7GESKX3GTA4PYB2HHIY4UB3KA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206118475 (579h3m52.07720151s) [noise=true , address=77.90.31.11:4555 , piece_id=VPILVCKGN2UK5RPLNNU3PQC72L7QN63ZXTG3435ZHMX45CWVH6DA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206425488 (579h3m52.077508523s) [noise=true , address=145.79.227.45:4229 , piece_id=C3CBOTVOIDTDCXJVNYUXQT5NREIOH5RA7EDD32LWS7WPIBHWYGMQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206745689 (579h3m52.077828734s) [noise=true , address=145.79.227.46:4602 , piece_id=6OJH335K6JBBPRIAL4QA5IWPSYN7SKEMPHR4ECWNSFDDVQQPD4TQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206758002 (579h3m52.077841037s) [noise=true , address=145.79.227.35:4483 , piece_id=ZJ7LEWE4ENTB2SHIXRPY3ASB6YHGJZ2BUJSJG4MYF53V6V4GFNGQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.207067291 (579h3m52.078150326s) [noise=true , address=45.91.116.9:4261 , piece_id=RAGXWVKLPDLNX2HNT34PUH6MZERHLCLSPMV75MQDF3MCCJEZQTIA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.207381760 (579h3m52.078464805s) [noise=true , address=145.79.227.44:4194 , piece_id=2Y43LHVPV53UAXRQOBBWTLAHR35AECGC3OE265TUMEGTD6MZYUEQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.207408040 (579h3m52.078491075s) [noise=true , address=145.79.227.5:4223 , piece_id=YVIFYEU2WJP6VP4YHQMOZV2SWPJLGLDJAWXMV352AC6WSDXMFVFA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.208987507 (579h3m52.080070542s) [noise=true , address=45.91.116.12:4227 , piece_id=CWF3YRREALI42IMYEPQZ6W3QWM6XFQKD2Z33DRUWVNEBLWS7OY7A, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:53.910451675 (579h3m53.78153472s) [upload=507625]: Next returned nil segment Dec 29 18:32:53.910474970 (579h3m53.781558005s) [upload=507625]: Waiting for error group managing segments... Dec 29 18:32:54.188945591 (579h3m54.060028626s) [noise=true , address=145.79.227.24:4581 , piece_id=MPFIHZZSEVR6ESIP5QSH5GGIFGE6KMCJ2DAR5ZSS66P66KDPAHFA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.196416346 (579h3m54.067499391s) [noise=true , address=145.79.227.45:4229 , piece_id=C3CBOTVOIDTDCXJVNYUXQT5NREIOH5RA7EDD32LWS7WPIBHWYGMQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.197993860 (579h3m54.069076895s) [noise=true , address=145.79.227.46:4602 , piece_id=6OJH335K6JBBPRIAL4QA5IWPSYN7SKEMPHR4ECWNSFDDVQQPD4TQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.200957113 (579h3m54.072040148s) [noise=true , address=145.79.227.22:4510 , piece_id=WIAUNELPEZP4OF64F4Z6IEGPDZM6ODUODHQFHEGJOFIG7KYKAFZA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.207922645 (579h3m54.07900569s) [noise=true , address=145.79.227.37:4565 , piece_id=KXKUL4642LGR5EFTFVVBOSYDTPDUCEZUNKUQS65KPQGDZDPVVV5Q, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.209264250 (579h3m54.080347285s) [noise=true , address=145.79.227.21:4475 , piece_id=JQD73PQDYUD4TZGIQBVSSZPNU5P7GESKX3GTA4PYB2HHIY4UB3KA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.211584358 (579h3m54.082667393s) [noise=true , address=145.79.227.5:4223 , piece_id=YVIFYEU2WJP6VP4YHQMOZV2SWPJLGLDJAWXMV352AC6WSDXMFVFA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.368210027 (579h3m54.239293062s) [noise=true , address=77.90.31.19:4842 , piece_id=2XCVPARINYJJP6Q5RL2ROS6YE2RJ4KR7WKCH2IY5XKH4VYLSP6NQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.431661819 (579h3m54.302744864s) [noise=true , address=77.90.31.11:4555 , piece_id=VPILVCKGN2UK5RPLNNU3PQC72L7QN63ZXTG3435ZHMX45CWVH6DA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.492327486 (579h3m54.363410521s) [noise=true , address=77.90.31.21:4880 , piece_id=BBZWEOHCQUZPZFYKH232RG6IFQEDA4TLNZ3HFRP2OHGMYIRMGSCA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.508530138 (579h3m54.379613173s) [noise=true , address=45.91.116.12:4227 , piece_id=CWF3YRREALI42IMYEPQZ6W3QWM6XFQKD2Z33DRUWVNEBLWS7OY7A, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.535358383 (579h3m54.406441418s) [noise=true , address=45.91.116.7:4009 , piece_id=VHAVEDXGATIR6DBXKM4XKUQMFLEJJLWVI4C4S46ZGCG6ZPB3V5CA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.579972706 (579h3m54.451055741s) [noise=true , address=45.91.116.9:4261 , piece_id=RAGXWVKLPDLNX2HNT34PUH6MZERHLCLSPMV75MQDF3MCCJEZQTIA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.580000098 (579h3m54.451083143s) [seg_pos={0 0}, upload=507625]: Segment reached optimal threshold of 13 pieces. Dec 29 18:32:54.580238622 (579h3m54.451321657s) [noise=true , address=145.79.227.44:4194 , piece_id=2Y43LHVPV53UAXRQOBBWTLAHR35AECGC3OE265TUMEGTD6MZYUEQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 18:32:54.580248180 (579h3m54.451331216s) [noise=true , address=145.79.227.44:4194 , piece_id=2Y43LHVPV53UAXRQOBBWTLAHR35AECGC3OE265TUMEGTD6MZYUEQ, seg_pos={0 0}, upload=507625]: Long tail cancellation due to optimal threshold Dec 29 18:32:54.580453021 (579h3m54.451536056s) [noise=true , address=77.90.31.14:4661 , piece_id=3QU3JVKHPZNTQPPRE6R5LPHO27JGARAESXVEIUXJ7CHHNEIGVKSA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 29 18:32:54.580459012 (579h3m54.451542047s) [noise=true , address=77.90.31.14:4661 , piece_id=3QU3JVKHPZNTQPPRE6R5LPHO27JGARAESXVEIUXJ7CHHNEIGVKSA, seg_pos={0 0}, upload=507625]: Long tail cancellation due to optimal threshold Dec 29 18:32:54.580481203 (579h3m54.451564259s) [noise=true , address=145.79.227.35:4483 , piece_id=ZJ7LEWE4ENTB2SHIXRPY3ASB6YHGJZ2BUJSJG4MYF53V6V4GFNGQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 29 18:32:54.580500681 (579h3m54.451583716s) [noise=true , address=145.79.227.35:4483 , piece_id=ZJ7LEWE4ENTB2SHIXRPY3ASB6YHGJZ2BUJSJG4MYF53V6V4GFNGQ, seg_pos={0 0}, upload=507625]: Long tail cancellation due to optimal threshold Dec 29 18:32:54.580533243 (579h3m54.451616288s) [seg_pos={0 0}, upload=507625]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:32:54.580611903 (579h3m54.451694938s) [upload=507625]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:32:54.580618224 (579h3m54.45170127s) [upload=507625]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:32:54.878900119 (579h3m54.749983154s) [upload=507625]: Done uploading segments... Dec 29 18:32:54.878920508 (579h3m54.750003553s) [upload=507625]: Upload finished. err: <nil> Duration: 12.896227183s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 39.839708345s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:35:41 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 39.83975321s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 3.304827025s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:35:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.304852423s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 50.761716066s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:35:01 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 50.761732817s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X.X....X.X..X.XXX..X.....X.X.....X...XX.X......XXX.XX.X.XX..] duration: 6.051829213s trace: https://jaeger.dev.storj.io/trace/6797d06f274d9c53 FAIL (2025-12-29 18:34:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: VHAVEDXGATIR6DBXKM4XKUQMFLEJJLWVI4C4S46ZGCG6ZPB3V5CA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: RAGXWVKLPDLNX2HNT34PUH6MZERHLCLSPMV75MQDF3MCCJEZQTIA) context deadline exceeded ======LOGS====== Dec 29 18:32:51.953251257 (579h3m51.824334302s) [upload=507625]: Starting upload Dec 29 18:32:51.953317824 (579h3m51.824400859s) [upload=507625]: Done starting upload Dec 29 18:32:51.953396002 (579h3m51.824479037s) [upload=507625]: Uploading segments... Dec 29 18:32:51.953668171 (579h3m51.824751216s) [upload=507625]: Got next segment. Inline: false Dec 29 18:32:51.953747421 (579h3m51.824830456s) [upload=507625]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:32:51.953752701 (579h3m51.824835746s) [upload=507625]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:32:52.202773254 (579h3m52.073856289s) [seg_pos={0 0}, upload=507625]: Begin upload segment... Dec 29 18:32:52.202827187 (579h3m52.073910222s) [seg_pos={0 0}, upload=507625]: Done begin upload segment. Dec 29 18:32:52.202859569 (579h3m52.073942604s) [noise=true , address=77.90.31.19:4842 , piece_id=2XCVPARINYJJP6Q5RL2ROS6YE2RJ4KR7WKCH2IY5XKH4VYLSP6NQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.202976842 (579h3m52.074059887s) [noise=true , address=77.90.31.14:4661 , piece_id=3QU3JVKHPZNTQPPRE6R5LPHO27JGARAESXVEIUXJ7CHHNEIGVKSA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.203661315 (579h3m52.07474435s) [noise=true , address=145.79.227.22:4510 , piece_id=WIAUNELPEZP4OF64F4Z6IEGPDZM6ODUODHQFHEGJOFIG7KYKAFZA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.203708254 (579h3m52.074791299s) [noise=true , address=145.79.227.24:4581 , piece_id=MPFIHZZSEVR6ESIP5QSH5GGIFGE6KMCJ2DAR5ZSS66P66KDPAHFA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.204243224 (579h3m52.075326269s) [noise=true , address=145.79.227.37:4565 , piece_id=KXKUL4642LGR5EFTFVVBOSYDTPDUCEZUNKUQS65KPQGDZDPVVV5Q, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.204825231 (579h3m52.075908276s) [noise=true , address=77.90.31.21:4880 , piece_id=BBZWEOHCQUZPZFYKH232RG6IFQEDA4TLNZ3HFRP2OHGMYIRMGSCA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.205446635 (579h3m52.07652967s) [noise=true , address=45.91.116.7:4009 , piece_id=VHAVEDXGATIR6DBXKM4XKUQMFLEJJLWVI4C4S46ZGCG6ZPB3V5CA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206012042 (579h3m52.077095077s) [noise=true , address=145.79.227.21:4475 , piece_id=JQD73PQDYUD4TZGIQBVSSZPNU5P7GESKX3GTA4PYB2HHIY4UB3KA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206118475 (579h3m52.07720151s) [noise=true , address=77.90.31.11:4555 , piece_id=VPILVCKGN2UK5RPLNNU3PQC72L7QN63ZXTG3435ZHMX45CWVH6DA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206425488 (579h3m52.077508523s) [noise=true , address=145.79.227.45:4229 , piece_id=C3CBOTVOIDTDCXJVNYUXQT5NREIOH5RA7EDD32LWS7WPIBHWYGMQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206745689 (579h3m52.077828734s) [noise=true , address=145.79.227.46:4602 , piece_id=6OJH335K6JBBPRIAL4QA5IWPSYN7SKEMPHR4ECWNSFDDVQQPD4TQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206758002 (579h3m52.077841037s) [noise=true , address=145.79.227.35:4483 , piece_id=ZJ7LEWE4ENTB2SHIXRPY3ASB6YHGJZ2BUJSJG4MYF53V6V4GFNGQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.207067291 (579h3m52.078150326s) [noise=true , address=45.91.116.9:4261 , piece_id=RAGXWVKLPDLNX2HNT34PUH6MZERHLCLSPMV75MQDF3MCCJEZQTIA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.207381760 (579h3m52.078464805s) [noise=true , address=145.79.227.44:4194 , piece_id=2Y43LHVPV53UAXRQOBBWTLAHR35AECGC3OE265TUMEGTD6MZYUEQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.207408040 (579h3m52.078491075s) [noise=true , address=145.79.227.5:4223 , piece_id=YVIFYEU2WJP6VP4YHQMOZV2SWPJLGLDJAWXMV352AC6WSDXMFVFA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.208987507 (579h3m52.080070542s) [noise=true , address=45.91.116.12:4227 , piece_id=CWF3YRREALI42IMYEPQZ6W3QWM6XFQKD2Z33DRUWVNEBLWS7OY7A, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:53.910451675 (579h3m53.78153472s) [upload=507625]: Next returned nil segment Dec 29 18:32:53.910474970 (579h3m53.781558005s) [upload=507625]: Waiting for error group managing segments... Dec 29 18:32:54.188945591 (579h3m54.060028626s) [noise=true , address=145.79.227.24:4581 , piece_id=MPFIHZZSEVR6ESIP5QSH5GGIFGE6KMCJ2DAR5ZSS66P66KDPAHFA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.196416346 (579h3m54.067499391s) [noise=true , address=145.79.227.45:4229 , piece_id=C3CBOTVOIDTDCXJVNYUXQT5NREIOH5RA7EDD32LWS7WPIBHWYGMQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.197993860 (579h3m54.069076895s) [noise=true , address=145.79.227.46:4602 , piece_id=6OJH335K6JBBPRIAL4QA5IWPSYN7SKEMPHR4ECWNSFDDVQQPD4TQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.200957113 (579h3m54.072040148s) [noise=true , address=145.79.227.22:4510 , piece_id=WIAUNELPEZP4OF64F4Z6IEGPDZM6ODUODHQFHEGJOFIG7KYKAFZA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.207922645 (579h3m54.07900569s) [noise=true , address=145.79.227.37:4565 , piece_id=KXKUL4642LGR5EFTFVVBOSYDTPDUCEZUNKUQS65KPQGDZDPVVV5Q, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.209264250 (579h3m54.080347285s) [noise=true , address=145.79.227.21:4475 , piece_id=JQD73PQDYUD4TZGIQBVSSZPNU5P7GESKX3GTA4PYB2HHIY4UB3KA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.211584358 (579h3m54.082667393s) [noise=true , address=145.79.227.5:4223 , piece_id=YVIFYEU2WJP6VP4YHQMOZV2SWPJLGLDJAWXMV352AC6WSDXMFVFA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.368210027 (579h3m54.239293062s) [noise=true , address=77.90.31.19:4842 , piece_id=2XCVPARINYJJP6Q5RL2ROS6YE2RJ4KR7WKCH2IY5XKH4VYLSP6NQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.431661819 (579h3m54.302744864s) [noise=true , address=77.90.31.11:4555 , piece_id=VPILVCKGN2UK5RPLNNU3PQC72L7QN63ZXTG3435ZHMX45CWVH6DA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.492327486 (579h3m54.363410521s) [noise=true , address=77.90.31.21:4880 , piece_id=BBZWEOHCQUZPZFYKH232RG6IFQEDA4TLNZ3HFRP2OHGMYIRMGSCA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.508530138 (579h3m54.379613173s) [noise=true , address=45.91.116.12:4227 , piece_id=CWF3YRREALI42IMYEPQZ6W3QWM6XFQKD2Z33DRUWVNEBLWS7OY7A, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.535358383 (579h3m54.406441418s) [noise=true , address=45.91.116.7:4009 , piece_id=VHAVEDXGATIR6DBXKM4XKUQMFLEJJLWVI4C4S46ZGCG6ZPB3V5CA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.579972706 (579h3m54.451055741s) [noise=true , address=45.91.116.9:4261 , piece_id=RAGXWVKLPDLNX2HNT34PUH6MZERHLCLSPMV75MQDF3MCCJEZQTIA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.580000098 (579h3m54.451083143s) [seg_pos={0 0}, upload=507625]: Segment reached optimal threshold of 13 pieces. Dec 29 18:32:54.580238622 (579h3m54.451321657s) [noise=true , address=145.79.227.44:4194 , piece_id=2Y43LHVPV53UAXRQOBBWTLAHR35AECGC3OE265TUMEGTD6MZYUEQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 18:32:54.580248180 (579h3m54.451331216s) [noise=true , address=145.79.227.44:4194 , piece_id=2Y43LHVPV53UAXRQOBBWTLAHR35AECGC3OE265TUMEGTD6MZYUEQ, seg_pos={0 0}, upload=507625]: Long tail cancellation due to optimal threshold Dec 29 18:32:54.580453021 (579h3m54.451536056s) [noise=true , address=77.90.31.14:4661 , piece_id=3QU3JVKHPZNTQPPRE6R5LPHO27JGARAESXVEIUXJ7CHHNEIGVKSA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 29 18:32:54.580459012 (579h3m54.451542047s) [noise=true , address=77.90.31.14:4661 , piece_id=3QU3JVKHPZNTQPPRE6R5LPHO27JGARAESXVEIUXJ7CHHNEIGVKSA, seg_pos={0 0}, upload=507625]: Long tail cancellation due to optimal threshold Dec 29 18:32:54.580481203 (579h3m54.451564259s) [noise=true , address=145.79.227.35:4483 , piece_id=ZJ7LEWE4ENTB2SHIXRPY3ASB6YHGJZ2BUJSJG4MYF53V6V4GFNGQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 29 18:32:54.580500681 (579h3m54.451583716s) [noise=true , address=145.79.227.35:4483 , piece_id=ZJ7LEWE4ENTB2SHIXRPY3ASB6YHGJZ2BUJSJG4MYF53V6V4GFNGQ, seg_pos={0 0}, upload=507625]: Long tail cancellation due to optimal threshold Dec 29 18:32:54.580533243 (579h3m54.451616288s) [seg_pos={0 0}, upload=507625]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:32:54.580611903 (579h3m54.451694938s) [upload=507625]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:32:54.580618224 (579h3m54.45170127s) [upload=507625]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:32:54.878900119 (579h3m54.749983154s) [upload=507625]: Done uploading segments... Dec 29 18:32:54.878920508 (579h3m54.750003553s) [upload=507625]: Upload finished. err: <nil> Duration: 6.051860233s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 3.452812209s trace: https://jaeger.dev.storj.io/trace/3a71004b134f364d FAIL (2025-12-29 18:34:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.452838571s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 36.516608596s trace: https://jaeger.dev.storj.io/trace/3965cb0ea03598d4 FAIL (2025-12-29 18:34:10 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396252632 (578h34m11.267335677s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396404001 (578h34m11.267487046s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.396756662 (578h34m11.267839697s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397056021 (578h34m11.268139076s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397232968 (578h34m11.268316013s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397730627 (578h34m11.268813672s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.397830697 (578h34m11.268913742s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398215290 (578h34m11.269298335s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398551349 (578h34m11.269634404s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.398722886 (578h34m11.269805921s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399155510 (578h34m11.270238555s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399161481 (578h34m11.270244526s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399200996 (578h34m11.270284031s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.399615726 (578h34m11.270698771s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:11.400058940 (578h34m11.271141975s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Uploading piece... Dec 29 18:03:12.774236760 (578h34m12.645319805s) [upload=507197]: Next returned nil segment Dec 29 18:03:12.774256168 (578h34m12.645339203s) [upload=507197]: Waiting for error group managing segments... Dec 29 18:03:13.159194658 (578h34m13.030277693s) [noise=true , address=64.49.13.139:4473 , piece_id=DJM2GPZNXXISIGXLEGBU7QZKOARPXQ3SWAS42H7P34RV6K6E6E2Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.248699118 (578h34m13.119782153s) [noise=true , address=145.79.227.45:4253 , piece_id=6KLJOLW6PMHZTWPAOU26GZOPVTMDGIJMYE7K2OO7JAP7O4KSJ3QQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.261598740 (578h34m13.132681775s) [noise=true , address=145.79.227.13:4831 , piece_id=EIFKQ3J5QIDNSIWHUIMDVQUXCPTKBHEAVJ7IP47OVFAEBDKEDISA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.278753206 (578h34m13.149836251s) [noise=true , address=145.79.227.6:4281 , piece_id=R2JI3X3N3QIC5477KFC66BOWT5DVXCGFG6AB537FGHJY2VZDW3AQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.282592536 (578h34m13.153675571s) [noise=true , address=145.79.227.7:4304 , piece_id=KB7XICGEFEHSBIS2KFJJPC6PK5Y2XLNNLTL4F3IL3ATZNICHKBSQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.412728334 (578h34m13.283811369s) [noise=true , address=77.90.31.17:4791 , piece_id=6ATM6FBGWSYDNE43IR27VDV3LRU7GD2FC6FJYRKH4BXFQVPIINCQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.468500613 (578h34m13.339583658s) [noise=true , address=77.90.31.21:4889 , piece_id=77UY2ZE7MZPWL2RKHPJRF52U3YLHDACBYVHSP73HYKBLPMICD5KQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.503034731 (578h34m13.374117776s) [noise=true , address=77.90.31.10:4523 , piece_id=JH5ZRWECHHLQRZLGUNCOPUSGKGHZXJZVUAMOWNV4S63S6SNFJMKQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.525424143 (578h34m13.396507188s) [noise=true , address=45.91.116.13:4185 , piece_id=2SZIZOQ6BX5563TQLRWIHJ6PLXP3TUYUVSQ4ENYWENUMPFHHHTDQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:13.531861651 (578h34m13.402944696s) [noise=true , address=77.90.31.14:4687 , piece_id=IZIEB25IFLUZU254I5VXCRQ7Q2D3OO77RTM7HQXY23WPIJFHWS7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.303933292 (578h34m14.175016338s) [noise=true , address=145.79.227.29:4955 , piece_id=N3RYMPZNSWRPREZUONNE7OKBDBXQQACHTKIHL7UHFFTZ5HKIMAPA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.399030752 (578h34m14.270113797s) [noise=true , address=145.79.227.43:4191 , piece_id=WAQ4353AOOIFBG63WLO6HQBCGMCUY4LCCVAAHXB3O62SHBAK6W7A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.456835942 (578h34m14.327918977s) [noise=true , address=145.79.227.14:4869 , piece_id=FQZAJ2VZ2UVNHKAEBTUVW55XJVOMRWOBJXLNWN3ZHEZGUWLZ6AYA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528103856 (578h34m14.399186911s) [noise=true , address=145.79.227.35:4485 , piece_id=LSEOPMF3PG46DH7Y5KKRJXCATI6QF52KSWRPGH6RTU7OVV7PGPMA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: <nil> Dec 29 18:03:14.528131469 (578h34m14.399214514s) [seg_pos={0 0}, upload=507197]: Segment reached optimal threshold of 14 pieces. Dec 29 18:03:14.528396945 (578h34m14.39948001s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fbxCfwkBdYcJG3LKJnU19Kyk9DtkhKdEAKoVfL353zC6Hge2k): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528407354 (578h34m14.3994904s) [noise=true , address=145.79.227.49:4361 , piece_id=CCINYEPPYQO7E6MVPLUGXZKSDW3MBTPQMBXQAVWGRZNX7HI2RMYQ, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528755948 (578h34m14.399838993s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.528765145 (578h34m14.399848191s) [noise=true , address=77.90.31.19:4862 , piece_id=WXQXV3AYHLFLTXNCBUIIUVINWJVYXK6QKRRXJAWGVJ6DHJHKV54Q, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.528937574 (578h34m14.400020619s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40120->145.79.227.3:4167: use of closed network connection Dec 29 18:03:14.528946110 (578h34m14.400029155s) [noise=true , address=145.79.227.3:4167 , piece_id=3G3CTCWSVIZU63MCYF6UUMLSMI4ZGXMOHAOE2NVLLLKL7GD3EGAA, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529091036 (578h34m14.400174081s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX): protocol: expected piece hash; context canceled; EOF Dec 29 18:03:14.529101576 (578h34m14.400184621s) [noise=true , address=77.90.31.12:4619 , piece_id=JV3KZN75QM336LDP3TTOMCW4YCH2P4A3B4INNHJZ4OJDAG3YK77A, seg_pos={0 0}, upload=507197]: Long tail cancellation due to optimal threshold Dec 29 18:03:14.529121614 (578h34m14.400204659s) [seg_pos={0 0}, upload=507197]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 18:03:14.529202960 (578h34m14.400286015s) [upload=507197]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:03:14.529209171 (578h34m14.400292216s) [upload=507197]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:03:14.842226543 (578h34m14.713309578s) [upload=507197]: Done uploading segments... Dec 29 18:03:14.842248945 (578h34m14.71333198s) [upload=507197]: Upload finished. err: <nil> Duration: 36.516635106s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XX.X....X.X..X.XXX..X.....X.X.....X...XX.X......XXX.XX.X.XX.] duration: 6.157398175s trace: https://jaeger.dev.storj.io/trace/6797d06f274d9c53 FAIL (2025-12-29 18:33:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: VHAVEDXGATIR6DBXKM4XKUQMFLEJJLWVI4C4S46ZGCG6ZPB3V5CA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: RAGXWVKLPDLNX2HNT34PUH6MZERHLCLSPMV75MQDF3MCCJEZQTIA) context deadline exceeded ======LOGS====== Dec 29 18:32:51.953251257 (579h3m51.824334302s) [upload=507625]: Starting upload Dec 29 18:32:51.953317824 (579h3m51.824400859s) [upload=507625]: Done starting upload Dec 29 18:32:51.953396002 (579h3m51.824479037s) [upload=507625]: Uploading segments... Dec 29 18:32:51.953668171 (579h3m51.824751216s) [upload=507625]: Got next segment. Inline: false Dec 29 18:32:51.953747421 (579h3m51.824830456s) [upload=507625]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 18:32:51.953752701 (579h3m51.824835746s) [upload=507625]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 18:32:52.202773254 (579h3m52.073856289s) [seg_pos={0 0}, upload=507625]: Begin upload segment... Dec 29 18:32:52.202827187 (579h3m52.073910222s) [seg_pos={0 0}, upload=507625]: Done begin upload segment. Dec 29 18:32:52.202859569 (579h3m52.073942604s) [noise=true , address=77.90.31.19:4842 , piece_id=2XCVPARINYJJP6Q5RL2ROS6YE2RJ4KR7WKCH2IY5XKH4VYLSP6NQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.202976842 (579h3m52.074059887s) [noise=true , address=77.90.31.14:4661 , piece_id=3QU3JVKHPZNTQPPRE6R5LPHO27JGARAESXVEIUXJ7CHHNEIGVKSA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.203661315 (579h3m52.07474435s) [noise=true , address=145.79.227.22:4510 , piece_id=WIAUNELPEZP4OF64F4Z6IEGPDZM6ODUODHQFHEGJOFIG7KYKAFZA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.203708254 (579h3m52.074791299s) [noise=true , address=145.79.227.24:4581 , piece_id=MPFIHZZSEVR6ESIP5QSH5GGIFGE6KMCJ2DAR5ZSS66P66KDPAHFA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.204243224 (579h3m52.075326269s) [noise=true , address=145.79.227.37:4565 , piece_id=KXKUL4642LGR5EFTFVVBOSYDTPDUCEZUNKUQS65KPQGDZDPVVV5Q, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.204825231 (579h3m52.075908276s) [noise=true , address=77.90.31.21:4880 , piece_id=BBZWEOHCQUZPZFYKH232RG6IFQEDA4TLNZ3HFRP2OHGMYIRMGSCA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.205446635 (579h3m52.07652967s) [noise=true , address=45.91.116.7:4009 , piece_id=VHAVEDXGATIR6DBXKM4XKUQMFLEJJLWVI4C4S46ZGCG6ZPB3V5CA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206012042 (579h3m52.077095077s) [noise=true , address=145.79.227.21:4475 , piece_id=JQD73PQDYUD4TZGIQBVSSZPNU5P7GESKX3GTA4PYB2HHIY4UB3KA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206118475 (579h3m52.07720151s) [noise=true , address=77.90.31.11:4555 , piece_id=VPILVCKGN2UK5RPLNNU3PQC72L7QN63ZXTG3435ZHMX45CWVH6DA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206425488 (579h3m52.077508523s) [noise=true , address=145.79.227.45:4229 , piece_id=C3CBOTVOIDTDCXJVNYUXQT5NREIOH5RA7EDD32LWS7WPIBHWYGMQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206745689 (579h3m52.077828734s) [noise=true , address=145.79.227.46:4602 , piece_id=6OJH335K6JBBPRIAL4QA5IWPSYN7SKEMPHR4ECWNSFDDVQQPD4TQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.206758002 (579h3m52.077841037s) [noise=true , address=145.79.227.35:4483 , piece_id=ZJ7LEWE4ENTB2SHIXRPY3ASB6YHGJZ2BUJSJG4MYF53V6V4GFNGQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.207067291 (579h3m52.078150326s) [noise=true , address=45.91.116.9:4261 , piece_id=RAGXWVKLPDLNX2HNT34PUH6MZERHLCLSPMV75MQDF3MCCJEZQTIA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.207381760 (579h3m52.078464805s) [noise=true , address=145.79.227.44:4194 , piece_id=2Y43LHVPV53UAXRQOBBWTLAHR35AECGC3OE265TUMEGTD6MZYUEQ, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.207408040 (579h3m52.078491075s) [noise=true , address=145.79.227.5:4223 , piece_id=YVIFYEU2WJP6VP4YHQMOZV2SWPJLGLDJAWXMV352AC6WSDXMFVFA, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:52.208987507 (579h3m52.080070542s) [noise=true , address=45.91.116.12:4227 , piece_id=CWF3YRREALI42IMYEPQZ6W3QWM6XFQKD2Z33DRUWVNEBLWS7OY7A, seg_pos={0 0}, upload=507625]: Uploading piece... Dec 29 18:32:53.910451675 (579h3m53.78153472s) [upload=507625]: Next returned nil segment Dec 29 18:32:53.910474970 (579h3m53.781558005s) [upload=507625]: Waiting for error group managing segments... Dec 29 18:32:54.188945591 (579h3m54.060028626s) [noise=true , address=145.79.227.24:4581 , piece_id=MPFIHZZSEVR6ESIP5QSH5GGIFGE6KMCJ2DAR5ZSS66P66KDPAHFA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.196416346 (579h3m54.067499391s) [noise=true , address=145.79.227.45:4229 , piece_id=C3CBOTVOIDTDCXJVNYUXQT5NREIOH5RA7EDD32LWS7WPIBHWYGMQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.197993860 (579h3m54.069076895s) [noise=true , address=145.79.227.46:4602 , piece_id=6OJH335K6JBBPRIAL4QA5IWPSYN7SKEMPHR4ECWNSFDDVQQPD4TQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.200957113 (579h3m54.072040148s) [noise=true , address=145.79.227.22:4510 , piece_id=WIAUNELPEZP4OF64F4Z6IEGPDZM6ODUODHQFHEGJOFIG7KYKAFZA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.207922645 (579h3m54.07900569s) [noise=true , address=145.79.227.37:4565 , piece_id=KXKUL4642LGR5EFTFVVBOSYDTPDUCEZUNKUQS65KPQGDZDPVVV5Q, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.209264250 (579h3m54.080347285s) [noise=true , address=145.79.227.21:4475 , piece_id=JQD73PQDYUD4TZGIQBVSSZPNU5P7GESKX3GTA4PYB2HHIY4UB3KA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.211584358 (579h3m54.082667393s) [noise=true , address=145.79.227.5:4223 , piece_id=YVIFYEU2WJP6VP4YHQMOZV2SWPJLGLDJAWXMV352AC6WSDXMFVFA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.368210027 (579h3m54.239293062s) [noise=true , address=77.90.31.19:4842 , piece_id=2XCVPARINYJJP6Q5RL2ROS6YE2RJ4KR7WKCH2IY5XKH4VYLSP6NQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.431661819 (579h3m54.302744864s) [noise=true , address=77.90.31.11:4555 , piece_id=VPILVCKGN2UK5RPLNNU3PQC72L7QN63ZXTG3435ZHMX45CWVH6DA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.492327486 (579h3m54.363410521s) [noise=true , address=77.90.31.21:4880 , piece_id=BBZWEOHCQUZPZFYKH232RG6IFQEDA4TLNZ3HFRP2OHGMYIRMGSCA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.508530138 (579h3m54.379613173s) [noise=true , address=45.91.116.12:4227 , piece_id=CWF3YRREALI42IMYEPQZ6W3QWM6XFQKD2Z33DRUWVNEBLWS7OY7A, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.535358383 (579h3m54.406441418s) [noise=true , address=45.91.116.7:4009 , piece_id=VHAVEDXGATIR6DBXKM4XKUQMFLEJJLWVI4C4S46ZGCG6ZPB3V5CA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.579972706 (579h3m54.451055741s) [noise=true , address=45.91.116.9:4261 , piece_id=RAGXWVKLPDLNX2HNT34PUH6MZERHLCLSPMV75MQDF3MCCJEZQTIA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: <nil> Dec 29 18:32:54.580000098 (579h3m54.451083143s) [seg_pos={0 0}, upload=507625]: Segment reached optimal threshold of 13 pieces. Dec 29 18:32:54.580238622 (579h3m54.451321657s) [noise=true , address=145.79.227.44:4194 , piece_id=2Y43LHVPV53UAXRQOBBWTLAHR35AECGC3OE265TUMEGTD6MZYUEQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 29 18:32:54.580248180 (579h3m54.451331216s) [noise=true , address=145.79.227.44:4194 , piece_id=2Y43LHVPV53UAXRQOBBWTLAHR35AECGC3OE265TUMEGTD6MZYUEQ, seg_pos={0 0}, upload=507625]: Long tail cancellation due to optimal threshold Dec 29 18:32:54.580453021 (579h3m54.451536056s) [noise=true , address=77.90.31.14:4661 , piece_id=3QU3JVKHPZNTQPPRE6R5LPHO27JGARAESXVEIUXJ7CHHNEIGVKSA, seg_pos={0 0}, upload=507625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 29 18:32:54.580459012 (579h3m54.451542047s) [noise=true , address=77.90.31.14:4661 , piece_id=3QU3JVKHPZNTQPPRE6R5LPHO27JGARAESXVEIUXJ7CHHNEIGVKSA, seg_pos={0 0}, upload=507625]: Long tail cancellation due to optimal threshold Dec 29 18:32:54.580481203 (579h3m54.451564259s) [noise=true , address=145.79.227.35:4483 , piece_id=ZJ7LEWE4ENTB2SHIXRPY3ASB6YHGJZ2BUJSJG4MYF53V6V4GFNGQ, seg_pos={0 0}, upload=507625]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 29 18:32:54.580500681 (579h3m54.451583716s) [noise=true , address=145.79.227.35:4483 , piece_id=ZJ7LEWE4ENTB2SHIXRPY3ASB6YHGJZ2BUJSJG4MYF53V6V4GFNGQ, seg_pos={0 0}, upload=507625]: Long tail cancellation due to optimal threshold Dec 29 18:32:54.580533243 (579h3m54.451616288s) [seg_pos={0 0}, upload=507625]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 18:32:54.580611903 (579h3m54.451694938s) [upload=507625]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 18:32:54.580618224 (579h3m54.45170127s) [upload=507625]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 18:32:54.878900119 (579h3m54.749983154s) [upload=507625]: Done uploading segments... Dec 29 18:32:54.878920508 (579h3m54.750003553s) [upload=507625]: Upload finished. err: <nil> Duration: 6.157426668s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)