Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 7.112419016s FAIL (2025-10-23 14:50:07 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 7.112448782s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X.........X.............................................] duration: 5.894478239s FAIL (2025-10-23 14:50:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:18:06.304157422 (720h46m51.766249905s) [upload=634034]: Starting upload Oct 23 14:18:06.304347725 (720h46m51.766440218s) [upload=634034]: Done starting upload Oct 23 14:18:06.304632076 (720h46m51.766724559s) [upload=634034]: Uploading segments... Oct 23 14:18:06.304690778 (720h46m51.766783261s) [upload=634034]: Got next segment. Inline: false Oct 23 14:18:06.304730654 (720h46m51.766823147s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:18:06.304737897 (720h46m51.766830371s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:18:06.560110142 (720h46m52.022202615s) [seg_pos={0 0}, upload=634034]: Begin upload segment... Oct 23 14:18:06.560170466 (720h46m52.022262949s) [seg_pos={0 0}, upload=634034]: Done begin upload segment. Oct 23 14:18:06.560196877 (720h46m52.02228936s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.560384274 (720h46m52.022476767s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561087734 (720h46m52.023180217s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561246465 (720h46m52.023338948s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561388856 (720h46m52.02348134s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561682246 (720h46m52.023774729s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561840206 (720h46m52.023932699s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562036520 (720h46m52.024129003s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562369515 (720h46m52.024461998s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562402808 (720h46m52.024495301s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562804713 (720h46m52.024897196s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562996849 (720h46m52.025089332s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563336766 (720h46m52.025429249s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563574799 (720h46m52.025667282s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564175863 (720h46m52.026268346s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564848324 (720h46m52.026940798s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:08.284021691 (720h46m53.746114164s) [upload=634034]: Next returned nil segment Oct 23 14:18:08.284044505 (720h46m53.746136978s) [upload=634034]: Waiting for error group managing segments... Oct 23 14:18:08.566598218 (720h46m54.028690691s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.594453381 (720h46m54.056545864s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.598953419 (720h46m54.061045892s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.602023124 (720h46m54.064115658s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625098415 (720h46m54.087190898s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625760095 (720h46m54.087852578s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.631004962 (720h46m54.093097445s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.635035678 (720h46m54.097128161s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.645903372 (720h46m54.107995845s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.711835221 (720h46m54.173927704s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.719309664 (720h46m54.181402147s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.758298877 (720h46m54.22039136s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819795110 (720h46m54.281887593s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819860023 (720h46m54.281952516s) [seg_pos={0 0}, upload=634034]: Segment reached optimal threshold of 13 pieces. Oct 23 14:18:08.820128314 (720h46m54.282220807s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection Oct 23 14:18:08.820140978 (720h46m54.282233461s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820198287 (720h46m54.28229077s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820209729 (720h46m54.282302222s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820445267 (720h46m54.28253775s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820509670 (720h46m54.282602153s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820531111 (720h46m54.282623594s) [seg_pos={0 0}, upload=634034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:18:08.820624719 (720h46m54.282717202s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:18:08.820630982 (720h46m54.282723465s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:18:09.121056837 (720h46m54.58314933s) [upload=634034]: Done uploading segments... Oct 23 14:18:09.121104197 (720h46m54.58319668s) [upload=634034]: Upload finished. err: <nil> Duration: 5.894508726s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X.........X............................................] duration: 5.971262061s FAIL (2025-10-23 14:49:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:18:06.304157422 (720h46m51.766249905s) [upload=634034]: Starting upload Oct 23 14:18:06.304347725 (720h46m51.766440218s) [upload=634034]: Done starting upload Oct 23 14:18:06.304632076 (720h46m51.766724559s) [upload=634034]: Uploading segments... Oct 23 14:18:06.304690778 (720h46m51.766783261s) [upload=634034]: Got next segment. Inline: false Oct 23 14:18:06.304730654 (720h46m51.766823147s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:18:06.304737897 (720h46m51.766830371s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:18:06.560110142 (720h46m52.022202615s) [seg_pos={0 0}, upload=634034]: Begin upload segment... Oct 23 14:18:06.560170466 (720h46m52.022262949s) [seg_pos={0 0}, upload=634034]: Done begin upload segment. Oct 23 14:18:06.560196877 (720h46m52.02228936s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.560384274 (720h46m52.022476767s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561087734 (720h46m52.023180217s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561246465 (720h46m52.023338948s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561388856 (720h46m52.02348134s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561682246 (720h46m52.023774729s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561840206 (720h46m52.023932699s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562036520 (720h46m52.024129003s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562369515 (720h46m52.024461998s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562402808 (720h46m52.024495301s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562804713 (720h46m52.024897196s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562996849 (720h46m52.025089332s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563336766 (720h46m52.025429249s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563574799 (720h46m52.025667282s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564175863 (720h46m52.026268346s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564848324 (720h46m52.026940798s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:08.284021691 (720h46m53.746114164s) [upload=634034]: Next returned nil segment Oct 23 14:18:08.284044505 (720h46m53.746136978s) [upload=634034]: Waiting for error group managing segments... Oct 23 14:18:08.566598218 (720h46m54.028690691s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.594453381 (720h46m54.056545864s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.598953419 (720h46m54.061045892s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.602023124 (720h46m54.064115658s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625098415 (720h46m54.087190898s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625760095 (720h46m54.087852578s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.631004962 (720h46m54.093097445s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.635035678 (720h46m54.097128161s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.645903372 (720h46m54.107995845s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.711835221 (720h46m54.173927704s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.719309664 (720h46m54.181402147s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.758298877 (720h46m54.22039136s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819795110 (720h46m54.281887593s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819860023 (720h46m54.281952516s) [seg_pos={0 0}, upload=634034]: Segment reached optimal threshold of 13 pieces. Oct 23 14:18:08.820128314 (720h46m54.282220807s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection Oct 23 14:18:08.820140978 (720h46m54.282233461s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820198287 (720h46m54.28229077s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820209729 (720h46m54.282302222s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820445267 (720h46m54.28253775s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820509670 (720h46m54.282602153s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820531111 (720h46m54.282623594s) [seg_pos={0 0}, upload=634034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:18:08.820624719 (720h46m54.282717202s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:18:08.820630982 (720h46m54.282723465s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:18:09.121056837 (720h46m54.58314933s) [upload=634034]: Done uploading segments... Oct 23 14:18:09.121104197 (720h46m54.58319668s) [upload=634034]: Upload finished. err: <nil> Duration: 5.971297978s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 2.032541687s FAIL (2025-10-23 14:49:28 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 2.032572515s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X.........X...........................................] duration: 5.942094729s FAIL (2025-10-23 14:49:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:18:06.304157422 (720h46m51.766249905s) [upload=634034]: Starting upload Oct 23 14:18:06.304347725 (720h46m51.766440218s) [upload=634034]: Done starting upload Oct 23 14:18:06.304632076 (720h46m51.766724559s) [upload=634034]: Uploading segments... Oct 23 14:18:06.304690778 (720h46m51.766783261s) [upload=634034]: Got next segment. Inline: false Oct 23 14:18:06.304730654 (720h46m51.766823147s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:18:06.304737897 (720h46m51.766830371s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:18:06.560110142 (720h46m52.022202615s) [seg_pos={0 0}, upload=634034]: Begin upload segment... Oct 23 14:18:06.560170466 (720h46m52.022262949s) [seg_pos={0 0}, upload=634034]: Done begin upload segment. Oct 23 14:18:06.560196877 (720h46m52.02228936s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.560384274 (720h46m52.022476767s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561087734 (720h46m52.023180217s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561246465 (720h46m52.023338948s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561388856 (720h46m52.02348134s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561682246 (720h46m52.023774729s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561840206 (720h46m52.023932699s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562036520 (720h46m52.024129003s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562369515 (720h46m52.024461998s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562402808 (720h46m52.024495301s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562804713 (720h46m52.024897196s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562996849 (720h46m52.025089332s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563336766 (720h46m52.025429249s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563574799 (720h46m52.025667282s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564175863 (720h46m52.026268346s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564848324 (720h46m52.026940798s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:08.284021691 (720h46m53.746114164s) [upload=634034]: Next returned nil segment Oct 23 14:18:08.284044505 (720h46m53.746136978s) [upload=634034]: Waiting for error group managing segments... Oct 23 14:18:08.566598218 (720h46m54.028690691s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.594453381 (720h46m54.056545864s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.598953419 (720h46m54.061045892s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.602023124 (720h46m54.064115658s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625098415 (720h46m54.087190898s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625760095 (720h46m54.087852578s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.631004962 (720h46m54.093097445s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.635035678 (720h46m54.097128161s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.645903372 (720h46m54.107995845s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.711835221 (720h46m54.173927704s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.719309664 (720h46m54.181402147s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.758298877 (720h46m54.22039136s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819795110 (720h46m54.281887593s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819860023 (720h46m54.281952516s) [seg_pos={0 0}, upload=634034]: Segment reached optimal threshold of 13 pieces. Oct 23 14:18:08.820128314 (720h46m54.282220807s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection Oct 23 14:18:08.820140978 (720h46m54.282233461s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820198287 (720h46m54.28229077s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820209729 (720h46m54.282302222s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820445267 (720h46m54.28253775s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820509670 (720h46m54.282602153s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820531111 (720h46m54.282623594s) [seg_pos={0 0}, upload=634034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:18:08.820624719 (720h46m54.282717202s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:18:08.820630982 (720h46m54.282723465s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:18:09.121056837 (720h46m54.58314933s) [upload=634034]: Done uploading segments... Oct 23 14:18:09.121104197 (720h46m54.58319668s) [upload=634034]: Upload finished. err: <nil> Duration: 5.942125929s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 6.301794508s FAIL (2025-10-23 14:49:11 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.301828522s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X.........X..........................................] duration: 5.899978135s FAIL (2025-10-23 14:48:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:18:06.304157422 (720h46m51.766249905s) [upload=634034]: Starting upload Oct 23 14:18:06.304347725 (720h46m51.766440218s) [upload=634034]: Done starting upload Oct 23 14:18:06.304632076 (720h46m51.766724559s) [upload=634034]: Uploading segments... Oct 23 14:18:06.304690778 (720h46m51.766783261s) [upload=634034]: Got next segment. Inline: false Oct 23 14:18:06.304730654 (720h46m51.766823147s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:18:06.304737897 (720h46m51.766830371s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:18:06.560110142 (720h46m52.022202615s) [seg_pos={0 0}, upload=634034]: Begin upload segment... Oct 23 14:18:06.560170466 (720h46m52.022262949s) [seg_pos={0 0}, upload=634034]: Done begin upload segment. Oct 23 14:18:06.560196877 (720h46m52.02228936s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.560384274 (720h46m52.022476767s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561087734 (720h46m52.023180217s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561246465 (720h46m52.023338948s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561388856 (720h46m52.02348134s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561682246 (720h46m52.023774729s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561840206 (720h46m52.023932699s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562036520 (720h46m52.024129003s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562369515 (720h46m52.024461998s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562402808 (720h46m52.024495301s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562804713 (720h46m52.024897196s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562996849 (720h46m52.025089332s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563336766 (720h46m52.025429249s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563574799 (720h46m52.025667282s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564175863 (720h46m52.026268346s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564848324 (720h46m52.026940798s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:08.284021691 (720h46m53.746114164s) [upload=634034]: Next returned nil segment Oct 23 14:18:08.284044505 (720h46m53.746136978s) [upload=634034]: Waiting for error group managing segments... Oct 23 14:18:08.566598218 (720h46m54.028690691s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.594453381 (720h46m54.056545864s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.598953419 (720h46m54.061045892s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.602023124 (720h46m54.064115658s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625098415 (720h46m54.087190898s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625760095 (720h46m54.087852578s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.631004962 (720h46m54.093097445s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.635035678 (720h46m54.097128161s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.645903372 (720h46m54.107995845s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.711835221 (720h46m54.173927704s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.719309664 (720h46m54.181402147s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.758298877 (720h46m54.22039136s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819795110 (720h46m54.281887593s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819860023 (720h46m54.281952516s) [seg_pos={0 0}, upload=634034]: Segment reached optimal threshold of 13 pieces. Oct 23 14:18:08.820128314 (720h46m54.282220807s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection Oct 23 14:18:08.820140978 (720h46m54.282233461s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820198287 (720h46m54.28229077s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820209729 (720h46m54.282302222s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820445267 (720h46m54.28253775s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820509670 (720h46m54.282602153s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820531111 (720h46m54.282623594s) [seg_pos={0 0}, upload=634034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:18:08.820624719 (720h46m54.282717202s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:18:08.820630982 (720h46m54.282723465s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:18:09.121056837 (720h46m54.58314933s) [upload=634034]: Done uploading segments... Oct 23 14:18:09.121104197 (720h46m54.58319668s) [upload=634034]: Upload finished. err: <nil> Duration: 5.900005578s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 1.994865454s FAIL (2025-10-23 14:48:42 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 1.994897516s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 6.096135206s FAIL (2025-10-23 14:48:41 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.096166915s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X.........X.........................................] duration: 5.943311657s FAIL (2025-10-23 14:48:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:18:06.304157422 (720h46m51.766249905s) [upload=634034]: Starting upload Oct 23 14:18:06.304347725 (720h46m51.766440218s) [upload=634034]: Done starting upload Oct 23 14:18:06.304632076 (720h46m51.766724559s) [upload=634034]: Uploading segments... Oct 23 14:18:06.304690778 (720h46m51.766783261s) [upload=634034]: Got next segment. Inline: false Oct 23 14:18:06.304730654 (720h46m51.766823147s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:18:06.304737897 (720h46m51.766830371s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:18:06.560110142 (720h46m52.022202615s) [seg_pos={0 0}, upload=634034]: Begin upload segment... Oct 23 14:18:06.560170466 (720h46m52.022262949s) [seg_pos={0 0}, upload=634034]: Done begin upload segment. Oct 23 14:18:06.560196877 (720h46m52.02228936s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.560384274 (720h46m52.022476767s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561087734 (720h46m52.023180217s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561246465 (720h46m52.023338948s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561388856 (720h46m52.02348134s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561682246 (720h46m52.023774729s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561840206 (720h46m52.023932699s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562036520 (720h46m52.024129003s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562369515 (720h46m52.024461998s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562402808 (720h46m52.024495301s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562804713 (720h46m52.024897196s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562996849 (720h46m52.025089332s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563336766 (720h46m52.025429249s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563574799 (720h46m52.025667282s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564175863 (720h46m52.026268346s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564848324 (720h46m52.026940798s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:08.284021691 (720h46m53.746114164s) [upload=634034]: Next returned nil segment Oct 23 14:18:08.284044505 (720h46m53.746136978s) [upload=634034]: Waiting for error group managing segments... Oct 23 14:18:08.566598218 (720h46m54.028690691s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.594453381 (720h46m54.056545864s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.598953419 (720h46m54.061045892s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.602023124 (720h46m54.064115658s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625098415 (720h46m54.087190898s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625760095 (720h46m54.087852578s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.631004962 (720h46m54.093097445s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.635035678 (720h46m54.097128161s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.645903372 (720h46m54.107995845s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.711835221 (720h46m54.173927704s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.719309664 (720h46m54.181402147s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.758298877 (720h46m54.22039136s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819795110 (720h46m54.281887593s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819860023 (720h46m54.281952516s) [seg_pos={0 0}, upload=634034]: Segment reached optimal threshold of 13 pieces. Oct 23 14:18:08.820128314 (720h46m54.282220807s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection Oct 23 14:18:08.820140978 (720h46m54.282233461s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820198287 (720h46m54.28229077s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820209729 (720h46m54.282302222s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820445267 (720h46m54.28253775s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820509670 (720h46m54.282602153s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820531111 (720h46m54.282623594s) [seg_pos={0 0}, upload=634034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:18:08.820624719 (720h46m54.282717202s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:18:08.820630982 (720h46m54.282723465s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:18:09.121056837 (720h46m54.58314933s) [upload=634034]: Done uploading segments... Oct 23 14:18:09.121104197 (720h46m54.58319668s) [upload=634034]: Upload finished. err: <nil> Duration: 5.94334s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.510877534s FAIL (2025-10-23 14:48:04 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.510911128s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 2.063802712s FAIL (2025-10-23 14:47:50 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 2.063858609s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 6.910871021s FAIL (2025-10-23 14:47:38 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.910901759s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 2.103081837s FAIL (2025-10-23 14:47:26 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 2.103111122s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X.........X........................................] duration: 5.867857756s FAIL (2025-10-23 14:47:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:18:06.304157422 (720h46m51.766249905s) [upload=634034]: Starting upload Oct 23 14:18:06.304347725 (720h46m51.766440218s) [upload=634034]: Done starting upload Oct 23 14:18:06.304632076 (720h46m51.766724559s) [upload=634034]: Uploading segments... Oct 23 14:18:06.304690778 (720h46m51.766783261s) [upload=634034]: Got next segment. Inline: false Oct 23 14:18:06.304730654 (720h46m51.766823147s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:18:06.304737897 (720h46m51.766830371s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:18:06.560110142 (720h46m52.022202615s) [seg_pos={0 0}, upload=634034]: Begin upload segment... Oct 23 14:18:06.560170466 (720h46m52.022262949s) [seg_pos={0 0}, upload=634034]: Done begin upload segment. Oct 23 14:18:06.560196877 (720h46m52.02228936s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.560384274 (720h46m52.022476767s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561087734 (720h46m52.023180217s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561246465 (720h46m52.023338948s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561388856 (720h46m52.02348134s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561682246 (720h46m52.023774729s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561840206 (720h46m52.023932699s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562036520 (720h46m52.024129003s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562369515 (720h46m52.024461998s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562402808 (720h46m52.024495301s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562804713 (720h46m52.024897196s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562996849 (720h46m52.025089332s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563336766 (720h46m52.025429249s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563574799 (720h46m52.025667282s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564175863 (720h46m52.026268346s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564848324 (720h46m52.026940798s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:08.284021691 (720h46m53.746114164s) [upload=634034]: Next returned nil segment Oct 23 14:18:08.284044505 (720h46m53.746136978s) [upload=634034]: Waiting for error group managing segments... Oct 23 14:18:08.566598218 (720h46m54.028690691s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.594453381 (720h46m54.056545864s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.598953419 (720h46m54.061045892s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.602023124 (720h46m54.064115658s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625098415 (720h46m54.087190898s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625760095 (720h46m54.087852578s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.631004962 (720h46m54.093097445s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.635035678 (720h46m54.097128161s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.645903372 (720h46m54.107995845s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.711835221 (720h46m54.173927704s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.719309664 (720h46m54.181402147s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.758298877 (720h46m54.22039136s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819795110 (720h46m54.281887593s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819860023 (720h46m54.281952516s) [seg_pos={0 0}, upload=634034]: Segment reached optimal threshold of 13 pieces. Oct 23 14:18:08.820128314 (720h46m54.282220807s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection Oct 23 14:18:08.820140978 (720h46m54.282233461s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820198287 (720h46m54.28229077s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820209729 (720h46m54.282302222s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820445267 (720h46m54.28253775s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820509670 (720h46m54.282602153s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820531111 (720h46m54.282623594s) [seg_pos={0 0}, upload=634034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:18:08.820624719 (720h46m54.282717202s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:18:08.820630982 (720h46m54.282723465s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:18:09.121056837 (720h46m54.58314933s) [upload=634034]: Done uploading segments... Oct 23 14:18:09.121104197 (720h46m54.58319668s) [upload=634034]: Upload finished. err: <nil> Duration: 5.867890929s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 2.047617561s FAIL (2025-10-23 14:47:08 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 2.047652989s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X.........X.......................................] duration: 6.106652715s FAIL (2025-10-23 14:46:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:18:06.304157422 (720h46m51.766249905s) [upload=634034]: Starting upload Oct 23 14:18:06.304347725 (720h46m51.766440218s) [upload=634034]: Done starting upload Oct 23 14:18:06.304632076 (720h46m51.766724559s) [upload=634034]: Uploading segments... Oct 23 14:18:06.304690778 (720h46m51.766783261s) [upload=634034]: Got next segment. Inline: false Oct 23 14:18:06.304730654 (720h46m51.766823147s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:18:06.304737897 (720h46m51.766830371s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:18:06.560110142 (720h46m52.022202615s) [seg_pos={0 0}, upload=634034]: Begin upload segment... Oct 23 14:18:06.560170466 (720h46m52.022262949s) [seg_pos={0 0}, upload=634034]: Done begin upload segment. Oct 23 14:18:06.560196877 (720h46m52.02228936s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.560384274 (720h46m52.022476767s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561087734 (720h46m52.023180217s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561246465 (720h46m52.023338948s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561388856 (720h46m52.02348134s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561682246 (720h46m52.023774729s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561840206 (720h46m52.023932699s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562036520 (720h46m52.024129003s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562369515 (720h46m52.024461998s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562402808 (720h46m52.024495301s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562804713 (720h46m52.024897196s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562996849 (720h46m52.025089332s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563336766 (720h46m52.025429249s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563574799 (720h46m52.025667282s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564175863 (720h46m52.026268346s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564848324 (720h46m52.026940798s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:08.284021691 (720h46m53.746114164s) [upload=634034]: Next returned nil segment Oct 23 14:18:08.284044505 (720h46m53.746136978s) [upload=634034]: Waiting for error group managing segments... Oct 23 14:18:08.566598218 (720h46m54.028690691s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.594453381 (720h46m54.056545864s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.598953419 (720h46m54.061045892s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.602023124 (720h46m54.064115658s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625098415 (720h46m54.087190898s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625760095 (720h46m54.087852578s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.631004962 (720h46m54.093097445s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.635035678 (720h46m54.097128161s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.645903372 (720h46m54.107995845s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.711835221 (720h46m54.173927704s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.719309664 (720h46m54.181402147s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.758298877 (720h46m54.22039136s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819795110 (720h46m54.281887593s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819860023 (720h46m54.281952516s) [seg_pos={0 0}, upload=634034]: Segment reached optimal threshold of 13 pieces. Oct 23 14:18:08.820128314 (720h46m54.282220807s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection Oct 23 14:18:08.820140978 (720h46m54.282233461s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820198287 (720h46m54.28229077s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820209729 (720h46m54.282302222s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820445267 (720h46m54.28253775s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820509670 (720h46m54.282602153s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820531111 (720h46m54.282623594s) [seg_pos={0 0}, upload=634034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:18:08.820624719 (720h46m54.282717202s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:18:08.820630982 (720h46m54.282723465s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:18:09.121056837 (720h46m54.58314933s) [upload=634034]: Done uploading segments... Oct 23 14:18:09.121104197 (720h46m54.58319668s) [upload=634034]: Upload finished. err: <nil> Duration: 6.106690867s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 10.718162592s FAIL (2025-10-23 14:46:39 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 10.71819835s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 2.896540695s FAIL (2025-10-23 14:46:11 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 2.896581512s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X.........X......................................] duration: 6.152766054s FAIL (2025-10-23 14:46:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:18:06.304157422 (720h46m51.766249905s) [upload=634034]: Starting upload Oct 23 14:18:06.304347725 (720h46m51.766440218s) [upload=634034]: Done starting upload Oct 23 14:18:06.304632076 (720h46m51.766724559s) [upload=634034]: Uploading segments... Oct 23 14:18:06.304690778 (720h46m51.766783261s) [upload=634034]: Got next segment. Inline: false Oct 23 14:18:06.304730654 (720h46m51.766823147s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:18:06.304737897 (720h46m51.766830371s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:18:06.560110142 (720h46m52.022202615s) [seg_pos={0 0}, upload=634034]: Begin upload segment... Oct 23 14:18:06.560170466 (720h46m52.022262949s) [seg_pos={0 0}, upload=634034]: Done begin upload segment. Oct 23 14:18:06.560196877 (720h46m52.02228936s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.560384274 (720h46m52.022476767s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561087734 (720h46m52.023180217s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561246465 (720h46m52.023338948s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561388856 (720h46m52.02348134s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561682246 (720h46m52.023774729s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561840206 (720h46m52.023932699s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562036520 (720h46m52.024129003s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562369515 (720h46m52.024461998s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562402808 (720h46m52.024495301s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562804713 (720h46m52.024897196s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562996849 (720h46m52.025089332s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563336766 (720h46m52.025429249s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563574799 (720h46m52.025667282s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564175863 (720h46m52.026268346s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564848324 (720h46m52.026940798s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:08.284021691 (720h46m53.746114164s) [upload=634034]: Next returned nil segment Oct 23 14:18:08.284044505 (720h46m53.746136978s) [upload=634034]: Waiting for error group managing segments... Oct 23 14:18:08.566598218 (720h46m54.028690691s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.594453381 (720h46m54.056545864s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.598953419 (720h46m54.061045892s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.602023124 (720h46m54.064115658s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625098415 (720h46m54.087190898s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625760095 (720h46m54.087852578s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.631004962 (720h46m54.093097445s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.635035678 (720h46m54.097128161s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.645903372 (720h46m54.107995845s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.711835221 (720h46m54.173927704s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.719309664 (720h46m54.181402147s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.758298877 (720h46m54.22039136s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819795110 (720h46m54.281887593s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819860023 (720h46m54.281952516s) [seg_pos={0 0}, upload=634034]: Segment reached optimal threshold of 13 pieces. Oct 23 14:18:08.820128314 (720h46m54.282220807s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection Oct 23 14:18:08.820140978 (720h46m54.282233461s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820198287 (720h46m54.28229077s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820209729 (720h46m54.282302222s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820445267 (720h46m54.28253775s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820509670 (720h46m54.282602153s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820531111 (720h46m54.282623594s) [seg_pos={0 0}, upload=634034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:18:08.820624719 (720h46m54.282717202s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:18:08.820630982 (720h46m54.282723465s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:18:09.121056837 (720h46m54.58314933s) [upload=634034]: Done uploading segments... Oct 23 14:18:09.121104197 (720h46m54.58319668s) [upload=634034]: Upload finished. err: <nil> Duration: 6.152802754s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.763026722s FAIL (2025-10-23 14:46:00 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.76306239s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 2.323276878s FAIL (2025-10-23 14:45:21 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 2.323309048s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X.........X.....................................] duration: 6.573295902s FAIL (2025-10-23 14:45:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:18:06.304157422 (720h46m51.766249905s) [upload=634034]: Starting upload Oct 23 14:18:06.304347725 (720h46m51.766440218s) [upload=634034]: Done starting upload Oct 23 14:18:06.304632076 (720h46m51.766724559s) [upload=634034]: Uploading segments... Oct 23 14:18:06.304690778 (720h46m51.766783261s) [upload=634034]: Got next segment. Inline: false Oct 23 14:18:06.304730654 (720h46m51.766823147s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:18:06.304737897 (720h46m51.766830371s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:18:06.560110142 (720h46m52.022202615s) [seg_pos={0 0}, upload=634034]: Begin upload segment... Oct 23 14:18:06.560170466 (720h46m52.022262949s) [seg_pos={0 0}, upload=634034]: Done begin upload segment. Oct 23 14:18:06.560196877 (720h46m52.02228936s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.560384274 (720h46m52.022476767s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561087734 (720h46m52.023180217s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561246465 (720h46m52.023338948s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561388856 (720h46m52.02348134s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561682246 (720h46m52.023774729s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561840206 (720h46m52.023932699s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562036520 (720h46m52.024129003s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562369515 (720h46m52.024461998s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562402808 (720h46m52.024495301s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562804713 (720h46m52.024897196s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562996849 (720h46m52.025089332s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563336766 (720h46m52.025429249s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563574799 (720h46m52.025667282s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564175863 (720h46m52.026268346s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564848324 (720h46m52.026940798s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:08.284021691 (720h46m53.746114164s) [upload=634034]: Next returned nil segment Oct 23 14:18:08.284044505 (720h46m53.746136978s) [upload=634034]: Waiting for error group managing segments... Oct 23 14:18:08.566598218 (720h46m54.028690691s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.594453381 (720h46m54.056545864s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.598953419 (720h46m54.061045892s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.602023124 (720h46m54.064115658s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625098415 (720h46m54.087190898s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625760095 (720h46m54.087852578s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.631004962 (720h46m54.093097445s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.635035678 (720h46m54.097128161s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.645903372 (720h46m54.107995845s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.711835221 (720h46m54.173927704s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.719309664 (720h46m54.181402147s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.758298877 (720h46m54.22039136s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819795110 (720h46m54.281887593s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819860023 (720h46m54.281952516s) [seg_pos={0 0}, upload=634034]: Segment reached optimal threshold of 13 pieces. Oct 23 14:18:08.820128314 (720h46m54.282220807s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection Oct 23 14:18:08.820140978 (720h46m54.282233461s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820198287 (720h46m54.28229077s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820209729 (720h46m54.282302222s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820445267 (720h46m54.28253775s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820509670 (720h46m54.282602153s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820531111 (720h46m54.282623594s) [seg_pos={0 0}, upload=634034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:18:08.820624719 (720h46m54.282717202s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:18:08.820630982 (720h46m54.282723465s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:18:09.121056837 (720h46m54.58314933s) [upload=634034]: Done uploading segments... Oct 23 14:18:09.121104197 (720h46m54.58319668s) [upload=634034]: Upload finished. err: <nil> Duration: 6.573324617s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 12.275176098s FAIL (2025-10-23 14:45:01 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 12.275214701s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 3.170354887s FAIL (2025-10-23 14:44:46 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 3.170390214s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 8.61221673s FAIL (2025-10-23 14:44:38 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 8.612246598s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X.........X....................................] duration: 6.160310764s FAIL (2025-10-23 14:44:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:18:06.304157422 (720h46m51.766249905s) [upload=634034]: Starting upload Oct 23 14:18:06.304347725 (720h46m51.766440218s) [upload=634034]: Done starting upload Oct 23 14:18:06.304632076 (720h46m51.766724559s) [upload=634034]: Uploading segments... Oct 23 14:18:06.304690778 (720h46m51.766783261s) [upload=634034]: Got next segment. Inline: false Oct 23 14:18:06.304730654 (720h46m51.766823147s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:18:06.304737897 (720h46m51.766830371s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:18:06.560110142 (720h46m52.022202615s) [seg_pos={0 0}, upload=634034]: Begin upload segment... Oct 23 14:18:06.560170466 (720h46m52.022262949s) [seg_pos={0 0}, upload=634034]: Done begin upload segment. Oct 23 14:18:06.560196877 (720h46m52.02228936s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.560384274 (720h46m52.022476767s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561087734 (720h46m52.023180217s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561246465 (720h46m52.023338948s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561388856 (720h46m52.02348134s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561682246 (720h46m52.023774729s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561840206 (720h46m52.023932699s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562036520 (720h46m52.024129003s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562369515 (720h46m52.024461998s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562402808 (720h46m52.024495301s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562804713 (720h46m52.024897196s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562996849 (720h46m52.025089332s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563336766 (720h46m52.025429249s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563574799 (720h46m52.025667282s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564175863 (720h46m52.026268346s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564848324 (720h46m52.026940798s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:08.284021691 (720h46m53.746114164s) [upload=634034]: Next returned nil segment Oct 23 14:18:08.284044505 (720h46m53.746136978s) [upload=634034]: Waiting for error group managing segments... Oct 23 14:18:08.566598218 (720h46m54.028690691s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.594453381 (720h46m54.056545864s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.598953419 (720h46m54.061045892s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.602023124 (720h46m54.064115658s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625098415 (720h46m54.087190898s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625760095 (720h46m54.087852578s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.631004962 (720h46m54.093097445s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.635035678 (720h46m54.097128161s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.645903372 (720h46m54.107995845s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.711835221 (720h46m54.173927704s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.719309664 (720h46m54.181402147s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.758298877 (720h46m54.22039136s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819795110 (720h46m54.281887593s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819860023 (720h46m54.281952516s) [seg_pos={0 0}, upload=634034]: Segment reached optimal threshold of 13 pieces. Oct 23 14:18:08.820128314 (720h46m54.282220807s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection Oct 23 14:18:08.820140978 (720h46m54.282233461s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820198287 (720h46m54.28229077s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820209729 (720h46m54.282302222s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820445267 (720h46m54.28253775s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820509670 (720h46m54.282602153s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820531111 (720h46m54.282623594s) [seg_pos={0 0}, upload=634034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:18:08.820624719 (720h46m54.282717202s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:18:08.820630982 (720h46m54.282723465s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:18:09.121056837 (720h46m54.58314933s) [upload=634034]: Done uploading segments... Oct 23 14:18:09.121104197 (720h46m54.58319668s) [upload=634034]: Upload finished. err: <nil> Duration: 6.160342977s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X.........X...................................] duration: 29.672862144s FAIL (2025-10-23 14:44:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:18:06.304157422 (720h46m51.766249905s) [upload=634034]: Starting upload Oct 23 14:18:06.304347725 (720h46m51.766440218s) [upload=634034]: Done starting upload Oct 23 14:18:06.304632076 (720h46m51.766724559s) [upload=634034]: Uploading segments... Oct 23 14:18:06.304690778 (720h46m51.766783261s) [upload=634034]: Got next segment. Inline: false Oct 23 14:18:06.304730654 (720h46m51.766823147s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:18:06.304737897 (720h46m51.766830371s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:18:06.560110142 (720h46m52.022202615s) [seg_pos={0 0}, upload=634034]: Begin upload segment... Oct 23 14:18:06.560170466 (720h46m52.022262949s) [seg_pos={0 0}, upload=634034]: Done begin upload segment. Oct 23 14:18:06.560196877 (720h46m52.02228936s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.560384274 (720h46m52.022476767s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561087734 (720h46m52.023180217s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561246465 (720h46m52.023338948s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561388856 (720h46m52.02348134s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561682246 (720h46m52.023774729s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561840206 (720h46m52.023932699s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562036520 (720h46m52.024129003s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562369515 (720h46m52.024461998s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562402808 (720h46m52.024495301s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562804713 (720h46m52.024897196s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562996849 (720h46m52.025089332s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563336766 (720h46m52.025429249s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563574799 (720h46m52.025667282s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564175863 (720h46m52.026268346s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564848324 (720h46m52.026940798s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:08.284021691 (720h46m53.746114164s) [upload=634034]: Next returned nil segment Oct 23 14:18:08.284044505 (720h46m53.746136978s) [upload=634034]: Waiting for error group managing segments... Oct 23 14:18:08.566598218 (720h46m54.028690691s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.594453381 (720h46m54.056545864s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.598953419 (720h46m54.061045892s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.602023124 (720h46m54.064115658s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625098415 (720h46m54.087190898s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625760095 (720h46m54.087852578s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.631004962 (720h46m54.093097445s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.635035678 (720h46m54.097128161s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.645903372 (720h46m54.107995845s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.711835221 (720h46m54.173927704s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.719309664 (720h46m54.181402147s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.758298877 (720h46m54.22039136s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819795110 (720h46m54.281887593s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819860023 (720h46m54.281952516s) [seg_pos={0 0}, upload=634034]: Segment reached optimal threshold of 13 pieces. Oct 23 14:18:08.820128314 (720h46m54.282220807s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection Oct 23 14:18:08.820140978 (720h46m54.282233461s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820198287 (720h46m54.28229077s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820209729 (720h46m54.282302222s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820445267 (720h46m54.28253775s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820509670 (720h46m54.282602153s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820531111 (720h46m54.282623594s) [seg_pos={0 0}, upload=634034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:18:08.820624719 (720h46m54.282717202s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:18:08.820630982 (720h46m54.282723465s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:18:09.121056837 (720h46m54.58314933s) [upload=634034]: Done uploading segments... Oct 23 14:18:09.121104197 (720h46m54.58319668s) [upload=634034]: Upload finished. err: <nil> Duration: 29.672889757s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 6.058425105s FAIL (2025-10-23 14:43:48 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.058474159s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 2.080444008s FAIL (2025-10-23 14:43:48 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 2.080483213s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 2.131035345s FAIL (2025-10-23 14:43:11 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 2.131064981s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.457296816s FAIL (2025-10-23 14:42:52 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.457321554s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X.........X..................................] duration: 5.917798495s FAIL (2025-10-23 14:42:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:18:06.304157422 (720h46m51.766249905s) [upload=634034]: Starting upload Oct 23 14:18:06.304347725 (720h46m51.766440218s) [upload=634034]: Done starting upload Oct 23 14:18:06.304632076 (720h46m51.766724559s) [upload=634034]: Uploading segments... Oct 23 14:18:06.304690778 (720h46m51.766783261s) [upload=634034]: Got next segment. Inline: false Oct 23 14:18:06.304730654 (720h46m51.766823147s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:18:06.304737897 (720h46m51.766830371s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:18:06.560110142 (720h46m52.022202615s) [seg_pos={0 0}, upload=634034]: Begin upload segment... Oct 23 14:18:06.560170466 (720h46m52.022262949s) [seg_pos={0 0}, upload=634034]: Done begin upload segment. Oct 23 14:18:06.560196877 (720h46m52.02228936s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.560384274 (720h46m52.022476767s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561087734 (720h46m52.023180217s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561246465 (720h46m52.023338948s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561388856 (720h46m52.02348134s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561682246 (720h46m52.023774729s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561840206 (720h46m52.023932699s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562036520 (720h46m52.024129003s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562369515 (720h46m52.024461998s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562402808 (720h46m52.024495301s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562804713 (720h46m52.024897196s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562996849 (720h46m52.025089332s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563336766 (720h46m52.025429249s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563574799 (720h46m52.025667282s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564175863 (720h46m52.026268346s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564848324 (720h46m52.026940798s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:08.284021691 (720h46m53.746114164s) [upload=634034]: Next returned nil segment Oct 23 14:18:08.284044505 (720h46m53.746136978s) [upload=634034]: Waiting for error group managing segments... Oct 23 14:18:08.566598218 (720h46m54.028690691s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.594453381 (720h46m54.056545864s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.598953419 (720h46m54.061045892s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.602023124 (720h46m54.064115658s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625098415 (720h46m54.087190898s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625760095 (720h46m54.087852578s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.631004962 (720h46m54.093097445s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.635035678 (720h46m54.097128161s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.645903372 (720h46m54.107995845s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.711835221 (720h46m54.173927704s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.719309664 (720h46m54.181402147s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.758298877 (720h46m54.22039136s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819795110 (720h46m54.281887593s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819860023 (720h46m54.281952516s) [seg_pos={0 0}, upload=634034]: Segment reached optimal threshold of 13 pieces. Oct 23 14:18:08.820128314 (720h46m54.282220807s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection Oct 23 14:18:08.820140978 (720h46m54.282233461s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820198287 (720h46m54.28229077s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820209729 (720h46m54.282302222s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820445267 (720h46m54.28253775s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820509670 (720h46m54.282602153s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820531111 (720h46m54.282623594s) [seg_pos={0 0}, upload=634034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:18:08.820624719 (720h46m54.282717202s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:18:08.820630982 (720h46m54.282723465s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:18:09.121056837 (720h46m54.58314933s) [upload=634034]: Done uploading segments... Oct 23 14:18:09.121104197 (720h46m54.58319668s) [upload=634034]: Upload finished. err: <nil> Duration: 5.91782222s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 1.992996772s FAIL (2025-10-23 14:42:09 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 1.993041749s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X.........X.................................] duration: 6.108974738s FAIL (2025-10-23 14:42:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:18:06.304157422 (720h46m51.766249905s) [upload=634034]: Starting upload Oct 23 14:18:06.304347725 (720h46m51.766440218s) [upload=634034]: Done starting upload Oct 23 14:18:06.304632076 (720h46m51.766724559s) [upload=634034]: Uploading segments... Oct 23 14:18:06.304690778 (720h46m51.766783261s) [upload=634034]: Got next segment. Inline: false Oct 23 14:18:06.304730654 (720h46m51.766823147s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:18:06.304737897 (720h46m51.766830371s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:18:06.560110142 (720h46m52.022202615s) [seg_pos={0 0}, upload=634034]: Begin upload segment... Oct 23 14:18:06.560170466 (720h46m52.022262949s) [seg_pos={0 0}, upload=634034]: Done begin upload segment. Oct 23 14:18:06.560196877 (720h46m52.02228936s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.560384274 (720h46m52.022476767s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561087734 (720h46m52.023180217s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561246465 (720h46m52.023338948s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561388856 (720h46m52.02348134s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561682246 (720h46m52.023774729s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561840206 (720h46m52.023932699s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562036520 (720h46m52.024129003s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562369515 (720h46m52.024461998s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562402808 (720h46m52.024495301s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562804713 (720h46m52.024897196s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562996849 (720h46m52.025089332s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563336766 (720h46m52.025429249s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563574799 (720h46m52.025667282s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564175863 (720h46m52.026268346s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564848324 (720h46m52.026940798s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:08.284021691 (720h46m53.746114164s) [upload=634034]: Next returned nil segment Oct 23 14:18:08.284044505 (720h46m53.746136978s) [upload=634034]: Waiting for error group managing segments... Oct 23 14:18:08.566598218 (720h46m54.028690691s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.594453381 (720h46m54.056545864s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.598953419 (720h46m54.061045892s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.602023124 (720h46m54.064115658s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625098415 (720h46m54.087190898s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625760095 (720h46m54.087852578s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.631004962 (720h46m54.093097445s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.635035678 (720h46m54.097128161s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.645903372 (720h46m54.107995845s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.711835221 (720h46m54.173927704s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.719309664 (720h46m54.181402147s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.758298877 (720h46m54.22039136s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819795110 (720h46m54.281887593s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819860023 (720h46m54.281952516s) [seg_pos={0 0}, upload=634034]: Segment reached optimal threshold of 13 pieces. Oct 23 14:18:08.820128314 (720h46m54.282220807s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection Oct 23 14:18:08.820140978 (720h46m54.282233461s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820198287 (720h46m54.28229077s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820209729 (720h46m54.282302222s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820445267 (720h46m54.28253775s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820509670 (720h46m54.282602153s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820531111 (720h46m54.282623594s) [seg_pos={0 0}, upload=634034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:18:08.820624719 (720h46m54.282717202s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:18:08.820630982 (720h46m54.282723465s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:18:09.121056837 (720h46m54.58314933s) [upload=634034]: Done uploading segments... Oct 23 14:18:09.121104197 (720h46m54.58319668s) [upload=634034]: Upload finished. err: <nil> Duration: 6.10901253s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 7.035207704s FAIL (2025-10-23 14:42:04 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 7.035285682s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.082907712s FAIL (2025-10-23 14:41:16 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 2.082940114s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 8.003868679s FAIL (2025-10-23 14:41:01 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 8.003907082s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X.........X................................] duration: 5.911483631s FAIL (2025-10-23 14:40:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:18:06.304157422 (720h46m51.766249905s) [upload=634034]: Starting upload Oct 23 14:18:06.304347725 (720h46m51.766440218s) [upload=634034]: Done starting upload Oct 23 14:18:06.304632076 (720h46m51.766724559s) [upload=634034]: Uploading segments... Oct 23 14:18:06.304690778 (720h46m51.766783261s) [upload=634034]: Got next segment. Inline: false Oct 23 14:18:06.304730654 (720h46m51.766823147s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:18:06.304737897 (720h46m51.766830371s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:18:06.560110142 (720h46m52.022202615s) [seg_pos={0 0}, upload=634034]: Begin upload segment... Oct 23 14:18:06.560170466 (720h46m52.022262949s) [seg_pos={0 0}, upload=634034]: Done begin upload segment. Oct 23 14:18:06.560196877 (720h46m52.02228936s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.560384274 (720h46m52.022476767s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561087734 (720h46m52.023180217s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561246465 (720h46m52.023338948s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561388856 (720h46m52.02348134s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561682246 (720h46m52.023774729s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561840206 (720h46m52.023932699s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562036520 (720h46m52.024129003s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562369515 (720h46m52.024461998s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562402808 (720h46m52.024495301s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562804713 (720h46m52.024897196s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562996849 (720h46m52.025089332s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563336766 (720h46m52.025429249s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563574799 (720h46m52.025667282s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564175863 (720h46m52.026268346s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564848324 (720h46m52.026940798s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:08.284021691 (720h46m53.746114164s) [upload=634034]: Next returned nil segment Oct 23 14:18:08.284044505 (720h46m53.746136978s) [upload=634034]: Waiting for error group managing segments... Oct 23 14:18:08.566598218 (720h46m54.028690691s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.594453381 (720h46m54.056545864s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.598953419 (720h46m54.061045892s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.602023124 (720h46m54.064115658s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625098415 (720h46m54.087190898s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625760095 (720h46m54.087852578s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.631004962 (720h46m54.093097445s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.635035678 (720h46m54.097128161s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.645903372 (720h46m54.107995845s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.711835221 (720h46m54.173927704s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.719309664 (720h46m54.181402147s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.758298877 (720h46m54.22039136s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819795110 (720h46m54.281887593s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819860023 (720h46m54.281952516s) [seg_pos={0 0}, upload=634034]: Segment reached optimal threshold of 13 pieces. Oct 23 14:18:08.820128314 (720h46m54.282220807s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection Oct 23 14:18:08.820140978 (720h46m54.282233461s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820198287 (720h46m54.28229077s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820209729 (720h46m54.282302222s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820445267 (720h46m54.28253775s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820509670 (720h46m54.282602153s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820531111 (720h46m54.282623594s) [seg_pos={0 0}, upload=634034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:18:08.820624719 (720h46m54.282717202s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:18:08.820630982 (720h46m54.282723465s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:18:09.121056837 (720h46m54.58314933s) [upload=634034]: Done uploading segments... Oct 23 14:18:09.121104197 (720h46m54.58319668s) [upload=634034]: Upload finished. err: <nil> Duration: 5.911517024s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X.........X...............................] duration: 6.001746219s FAIL (2025-10-23 14:40:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:18:06.304157422 (720h46m51.766249905s) [upload=634034]: Starting upload Oct 23 14:18:06.304347725 (720h46m51.766440218s) [upload=634034]: Done starting upload Oct 23 14:18:06.304632076 (720h46m51.766724559s) [upload=634034]: Uploading segments... Oct 23 14:18:06.304690778 (720h46m51.766783261s) [upload=634034]: Got next segment. Inline: false Oct 23 14:18:06.304730654 (720h46m51.766823147s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:18:06.304737897 (720h46m51.766830371s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:18:06.560110142 (720h46m52.022202615s) [seg_pos={0 0}, upload=634034]: Begin upload segment... Oct 23 14:18:06.560170466 (720h46m52.022262949s) [seg_pos={0 0}, upload=634034]: Done begin upload segment. Oct 23 14:18:06.560196877 (720h46m52.02228936s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.560384274 (720h46m52.022476767s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561087734 (720h46m52.023180217s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561246465 (720h46m52.023338948s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561388856 (720h46m52.02348134s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561682246 (720h46m52.023774729s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561840206 (720h46m52.023932699s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562036520 (720h46m52.024129003s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562369515 (720h46m52.024461998s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562402808 (720h46m52.024495301s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562804713 (720h46m52.024897196s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562996849 (720h46m52.025089332s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563336766 (720h46m52.025429249s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563574799 (720h46m52.025667282s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564175863 (720h46m52.026268346s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564848324 (720h46m52.026940798s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:08.284021691 (720h46m53.746114164s) [upload=634034]: Next returned nil segment Oct 23 14:18:08.284044505 (720h46m53.746136978s) [upload=634034]: Waiting for error group managing segments... Oct 23 14:18:08.566598218 (720h46m54.028690691s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.594453381 (720h46m54.056545864s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.598953419 (720h46m54.061045892s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.602023124 (720h46m54.064115658s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625098415 (720h46m54.087190898s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625760095 (720h46m54.087852578s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.631004962 (720h46m54.093097445s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.635035678 (720h46m54.097128161s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.645903372 (720h46m54.107995845s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.711835221 (720h46m54.173927704s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.719309664 (720h46m54.181402147s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.758298877 (720h46m54.22039136s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819795110 (720h46m54.281887593s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819860023 (720h46m54.281952516s) [seg_pos={0 0}, upload=634034]: Segment reached optimal threshold of 13 pieces. Oct 23 14:18:08.820128314 (720h46m54.282220807s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection Oct 23 14:18:08.820140978 (720h46m54.282233461s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820198287 (720h46m54.28229077s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820209729 (720h46m54.282302222s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820445267 (720h46m54.28253775s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820509670 (720h46m54.282602153s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820531111 (720h46m54.282623594s) [seg_pos={0 0}, upload=634034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:18:08.820624719 (720h46m54.282717202s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:18:08.820630982 (720h46m54.282723465s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:18:09.121056837 (720h46m54.58314933s) [upload=634034]: Done uploading segments... Oct 23 14:18:09.121104197 (720h46m54.58319668s) [upload=634034]: Upload finished. err: <nil> Duration: 6.001776709s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.01822411s FAIL (2025-10-23 14:40:28 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 2.018258295s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 7.106100111s FAIL (2025-10-23 14:40:24 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 7.106132884s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X.........X..............................] duration: 5.996078029s FAIL (2025-10-23 14:40:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:18:06.304157422 (720h46m51.766249905s) [upload=634034]: Starting upload Oct 23 14:18:06.304347725 (720h46m51.766440218s) [upload=634034]: Done starting upload Oct 23 14:18:06.304632076 (720h46m51.766724559s) [upload=634034]: Uploading segments... Oct 23 14:18:06.304690778 (720h46m51.766783261s) [upload=634034]: Got next segment. Inline: false Oct 23 14:18:06.304730654 (720h46m51.766823147s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:18:06.304737897 (720h46m51.766830371s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:18:06.560110142 (720h46m52.022202615s) [seg_pos={0 0}, upload=634034]: Begin upload segment... Oct 23 14:18:06.560170466 (720h46m52.022262949s) [seg_pos={0 0}, upload=634034]: Done begin upload segment. Oct 23 14:18:06.560196877 (720h46m52.02228936s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.560384274 (720h46m52.022476767s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561087734 (720h46m52.023180217s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561246465 (720h46m52.023338948s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561388856 (720h46m52.02348134s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561682246 (720h46m52.023774729s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561840206 (720h46m52.023932699s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562036520 (720h46m52.024129003s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562369515 (720h46m52.024461998s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562402808 (720h46m52.024495301s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562804713 (720h46m52.024897196s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562996849 (720h46m52.025089332s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563336766 (720h46m52.025429249s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563574799 (720h46m52.025667282s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564175863 (720h46m52.026268346s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564848324 (720h46m52.026940798s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:08.284021691 (720h46m53.746114164s) [upload=634034]: Next returned nil segment Oct 23 14:18:08.284044505 (720h46m53.746136978s) [upload=634034]: Waiting for error group managing segments... Oct 23 14:18:08.566598218 (720h46m54.028690691s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.594453381 (720h46m54.056545864s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.598953419 (720h46m54.061045892s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.602023124 (720h46m54.064115658s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625098415 (720h46m54.087190898s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625760095 (720h46m54.087852578s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.631004962 (720h46m54.093097445s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.635035678 (720h46m54.097128161s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.645903372 (720h46m54.107995845s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.711835221 (720h46m54.173927704s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.719309664 (720h46m54.181402147s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.758298877 (720h46m54.22039136s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819795110 (720h46m54.281887593s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819860023 (720h46m54.281952516s) [seg_pos={0 0}, upload=634034]: Segment reached optimal threshold of 13 pieces. Oct 23 14:18:08.820128314 (720h46m54.282220807s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection Oct 23 14:18:08.820140978 (720h46m54.282233461s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820198287 (720h46m54.28229077s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820209729 (720h46m54.282302222s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820445267 (720h46m54.28253775s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820509670 (720h46m54.282602153s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820531111 (720h46m54.282623594s) [seg_pos={0 0}, upload=634034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:18:08.820624719 (720h46m54.282717202s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:18:08.820630982 (720h46m54.282723465s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:18:09.121056837 (720h46m54.58314933s) [upload=634034]: Done uploading segments... Oct 23 14:18:09.121104197 (720h46m54.58319668s) [upload=634034]: Upload finished. err: <nil> Duration: 5.996114158s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X.........X.............................] duration: 5.876923131s FAIL (2025-10-23 14:39:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:18:06.304157422 (720h46m51.766249905s) [upload=634034]: Starting upload Oct 23 14:18:06.304347725 (720h46m51.766440218s) [upload=634034]: Done starting upload Oct 23 14:18:06.304632076 (720h46m51.766724559s) [upload=634034]: Uploading segments... Oct 23 14:18:06.304690778 (720h46m51.766783261s) [upload=634034]: Got next segment. Inline: false Oct 23 14:18:06.304730654 (720h46m51.766823147s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:18:06.304737897 (720h46m51.766830371s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:18:06.560110142 (720h46m52.022202615s) [seg_pos={0 0}, upload=634034]: Begin upload segment... Oct 23 14:18:06.560170466 (720h46m52.022262949s) [seg_pos={0 0}, upload=634034]: Done begin upload segment. Oct 23 14:18:06.560196877 (720h46m52.02228936s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.560384274 (720h46m52.022476767s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561087734 (720h46m52.023180217s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561246465 (720h46m52.023338948s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561388856 (720h46m52.02348134s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561682246 (720h46m52.023774729s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561840206 (720h46m52.023932699s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562036520 (720h46m52.024129003s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562369515 (720h46m52.024461998s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562402808 (720h46m52.024495301s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562804713 (720h46m52.024897196s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562996849 (720h46m52.025089332s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563336766 (720h46m52.025429249s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563574799 (720h46m52.025667282s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564175863 (720h46m52.026268346s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564848324 (720h46m52.026940798s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:08.284021691 (720h46m53.746114164s) [upload=634034]: Next returned nil segment Oct 23 14:18:08.284044505 (720h46m53.746136978s) [upload=634034]: Waiting for error group managing segments... Oct 23 14:18:08.566598218 (720h46m54.028690691s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.594453381 (720h46m54.056545864s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.598953419 (720h46m54.061045892s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.602023124 (720h46m54.064115658s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625098415 (720h46m54.087190898s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625760095 (720h46m54.087852578s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.631004962 (720h46m54.093097445s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.635035678 (720h46m54.097128161s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.645903372 (720h46m54.107995845s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.711835221 (720h46m54.173927704s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.719309664 (720h46m54.181402147s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.758298877 (720h46m54.22039136s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819795110 (720h46m54.281887593s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819860023 (720h46m54.281952516s) [seg_pos={0 0}, upload=634034]: Segment reached optimal threshold of 13 pieces. Oct 23 14:18:08.820128314 (720h46m54.282220807s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection Oct 23 14:18:08.820140978 (720h46m54.282233461s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820198287 (720h46m54.28229077s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820209729 (720h46m54.282302222s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820445267 (720h46m54.28253775s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820509670 (720h46m54.282602153s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820531111 (720h46m54.282623594s) [seg_pos={0 0}, upload=634034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:18:08.820624719 (720h46m54.282717202s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:18:08.820630982 (720h46m54.282723465s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:18:09.121056837 (720h46m54.58314933s) [upload=634034]: Done uploading segments... Oct 23 14:18:09.121104197 (720h46m54.58319668s) [upload=634034]: Upload finished. err: <nil> Duration: 5.876980542s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.001480658s FAIL (2025-10-23 14:39:20 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 2.00150307s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 7.091334292s FAIL (2025-10-23 14:39:18 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 7.091360792s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 2.052810708s FAIL (2025-10-23 14:39:02 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 2.052849452s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.270238583s FAIL (2025-10-23 14:38:58 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.270270314s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 6.378015777s FAIL (2025-10-23 14:38:38 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.378049912s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X.........X............................] duration: 5.988319623s FAIL (2025-10-23 14:38:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:18:06.304157422 (720h46m51.766249905s) [upload=634034]: Starting upload Oct 23 14:18:06.304347725 (720h46m51.766440218s) [upload=634034]: Done starting upload Oct 23 14:18:06.304632076 (720h46m51.766724559s) [upload=634034]: Uploading segments... Oct 23 14:18:06.304690778 (720h46m51.766783261s) [upload=634034]: Got next segment. Inline: false Oct 23 14:18:06.304730654 (720h46m51.766823147s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:18:06.304737897 (720h46m51.766830371s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:18:06.560110142 (720h46m52.022202615s) [seg_pos={0 0}, upload=634034]: Begin upload segment... Oct 23 14:18:06.560170466 (720h46m52.022262949s) [seg_pos={0 0}, upload=634034]: Done begin upload segment. Oct 23 14:18:06.560196877 (720h46m52.02228936s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.560384274 (720h46m52.022476767s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561087734 (720h46m52.023180217s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561246465 (720h46m52.023338948s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561388856 (720h46m52.02348134s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561682246 (720h46m52.023774729s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561840206 (720h46m52.023932699s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562036520 (720h46m52.024129003s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562369515 (720h46m52.024461998s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562402808 (720h46m52.024495301s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562804713 (720h46m52.024897196s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562996849 (720h46m52.025089332s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563336766 (720h46m52.025429249s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563574799 (720h46m52.025667282s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564175863 (720h46m52.026268346s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564848324 (720h46m52.026940798s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:08.284021691 (720h46m53.746114164s) [upload=634034]: Next returned nil segment Oct 23 14:18:08.284044505 (720h46m53.746136978s) [upload=634034]: Waiting for error group managing segments... Oct 23 14:18:08.566598218 (720h46m54.028690691s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.594453381 (720h46m54.056545864s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.598953419 (720h46m54.061045892s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.602023124 (720h46m54.064115658s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625098415 (720h46m54.087190898s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625760095 (720h46m54.087852578s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.631004962 (720h46m54.093097445s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.635035678 (720h46m54.097128161s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.645903372 (720h46m54.107995845s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.711835221 (720h46m54.173927704s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.719309664 (720h46m54.181402147s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.758298877 (720h46m54.22039136s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819795110 (720h46m54.281887593s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819860023 (720h46m54.281952516s) [seg_pos={0 0}, upload=634034]: Segment reached optimal threshold of 13 pieces. Oct 23 14:18:08.820128314 (720h46m54.282220807s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection Oct 23 14:18:08.820140978 (720h46m54.282233461s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820198287 (720h46m54.28229077s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820209729 (720h46m54.282302222s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820445267 (720h46m54.28253775s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820509670 (720h46m54.282602153s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820531111 (720h46m54.282623594s) [seg_pos={0 0}, upload=634034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:18:08.820624719 (720h46m54.282717202s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:18:08.820630982 (720h46m54.282723465s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:18:09.121056837 (720h46m54.58314933s) [upload=634034]: Done uploading segments... Oct 23 14:18:09.121104197 (720h46m54.58319668s) [upload=634034]: Upload finished. err: <nil> Duration: 5.988340502s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)