Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X.........X...........] duration: 10.536247994s FAIL (2025-10-23 14:26:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:18:06.304157422 (720h46m51.766249905s) [upload=634034]: Starting upload Oct 23 14:18:06.304347725 (720h46m51.766440218s) [upload=634034]: Done starting upload Oct 23 14:18:06.304632076 (720h46m51.766724559s) [upload=634034]: Uploading segments... Oct 23 14:18:06.304690778 (720h46m51.766783261s) [upload=634034]: Got next segment. Inline: false Oct 23 14:18:06.304730654 (720h46m51.766823147s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:18:06.304737897 (720h46m51.766830371s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:18:06.560110142 (720h46m52.022202615s) [seg_pos={0 0}, upload=634034]: Begin upload segment... Oct 23 14:18:06.560170466 (720h46m52.022262949s) [seg_pos={0 0}, upload=634034]: Done begin upload segment. Oct 23 14:18:06.560196877 (720h46m52.02228936s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.560384274 (720h46m52.022476767s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561087734 (720h46m52.023180217s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561246465 (720h46m52.023338948s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561388856 (720h46m52.02348134s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561682246 (720h46m52.023774729s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561840206 (720h46m52.023932699s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562036520 (720h46m52.024129003s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562369515 (720h46m52.024461998s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562402808 (720h46m52.024495301s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562804713 (720h46m52.024897196s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562996849 (720h46m52.025089332s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563336766 (720h46m52.025429249s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563574799 (720h46m52.025667282s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564175863 (720h46m52.026268346s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564848324 (720h46m52.026940798s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:08.284021691 (720h46m53.746114164s) [upload=634034]: Next returned nil segment Oct 23 14:18:08.284044505 (720h46m53.746136978s) [upload=634034]: Waiting for error group managing segments... Oct 23 14:18:08.566598218 (720h46m54.028690691s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.594453381 (720h46m54.056545864s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.598953419 (720h46m54.061045892s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.602023124 (720h46m54.064115658s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625098415 (720h46m54.087190898s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625760095 (720h46m54.087852578s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.631004962 (720h46m54.093097445s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.635035678 (720h46m54.097128161s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.645903372 (720h46m54.107995845s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.711835221 (720h46m54.173927704s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.719309664 (720h46m54.181402147s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.758298877 (720h46m54.22039136s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819795110 (720h46m54.281887593s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819860023 (720h46m54.281952516s) [seg_pos={0 0}, upload=634034]: Segment reached optimal threshold of 13 pieces. Oct 23 14:18:08.820128314 (720h46m54.282220807s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection Oct 23 14:18:08.820140978 (720h46m54.282233461s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820198287 (720h46m54.28229077s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820209729 (720h46m54.282302222s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820445267 (720h46m54.28253775s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820509670 (720h46m54.282602153s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820531111 (720h46m54.282623594s) [seg_pos={0 0}, upload=634034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:18:08.820624719 (720h46m54.282717202s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:18:08.820630982 (720h46m54.282723465s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:18:09.121056837 (720h46m54.58314933s) [upload=634034]: Done uploading segments... Oct 23 14:18:09.121104197 (720h46m54.58319668s) [upload=634034]: Upload finished. err: <nil> Duration: 10.536278241s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.60261437s FAIL (2025-10-23 14:26:02 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 2.602648936s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 10.478835643s FAIL (2025-10-23 14:25:51 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 10.478868205s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X.........X..........] duration: 8.959670634s FAIL (2025-10-23 14:25:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:18:06.304157422 (720h46m51.766249905s) [upload=634034]: Starting upload Oct 23 14:18:06.304347725 (720h46m51.766440218s) [upload=634034]: Done starting upload Oct 23 14:18:06.304632076 (720h46m51.766724559s) [upload=634034]: Uploading segments... Oct 23 14:18:06.304690778 (720h46m51.766783261s) [upload=634034]: Got next segment. Inline: false Oct 23 14:18:06.304730654 (720h46m51.766823147s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:18:06.304737897 (720h46m51.766830371s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:18:06.560110142 (720h46m52.022202615s) [seg_pos={0 0}, upload=634034]: Begin upload segment... Oct 23 14:18:06.560170466 (720h46m52.022262949s) [seg_pos={0 0}, upload=634034]: Done begin upload segment. Oct 23 14:18:06.560196877 (720h46m52.02228936s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.560384274 (720h46m52.022476767s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561087734 (720h46m52.023180217s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561246465 (720h46m52.023338948s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561388856 (720h46m52.02348134s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561682246 (720h46m52.023774729s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561840206 (720h46m52.023932699s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562036520 (720h46m52.024129003s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562369515 (720h46m52.024461998s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562402808 (720h46m52.024495301s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562804713 (720h46m52.024897196s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562996849 (720h46m52.025089332s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563336766 (720h46m52.025429249s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563574799 (720h46m52.025667282s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564175863 (720h46m52.026268346s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564848324 (720h46m52.026940798s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:08.284021691 (720h46m53.746114164s) [upload=634034]: Next returned nil segment Oct 23 14:18:08.284044505 (720h46m53.746136978s) [upload=634034]: Waiting for error group managing segments... Oct 23 14:18:08.566598218 (720h46m54.028690691s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.594453381 (720h46m54.056545864s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.598953419 (720h46m54.061045892s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.602023124 (720h46m54.064115658s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625098415 (720h46m54.087190898s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625760095 (720h46m54.087852578s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.631004962 (720h46m54.093097445s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.635035678 (720h46m54.097128161s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.645903372 (720h46m54.107995845s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.711835221 (720h46m54.173927704s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.719309664 (720h46m54.181402147s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.758298877 (720h46m54.22039136s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819795110 (720h46m54.281887593s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819860023 (720h46m54.281952516s) [seg_pos={0 0}, upload=634034]: Segment reached optimal threshold of 13 pieces. Oct 23 14:18:08.820128314 (720h46m54.282220807s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection Oct 23 14:18:08.820140978 (720h46m54.282233461s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820198287 (720h46m54.28229077s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820209729 (720h46m54.282302222s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820445267 (720h46m54.28253775s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820509670 (720h46m54.282602153s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820531111 (720h46m54.282623594s) [seg_pos={0 0}, upload=634034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:18:08.820624719 (720h46m54.282717202s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:18:08.820630982 (720h46m54.282723465s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:18:09.121056837 (720h46m54.58314933s) [upload=634034]: Done uploading segments... Oct 23 14:18:09.121104197 (720h46m54.58319668s) [upload=634034]: Upload finished. err: <nil> Duration: 8.95970969s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 4.66195646s FAIL (2025-10-23 14:25:13 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 4.662027184s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 7.21555643s FAIL (2025-10-23 14:24:56 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 7.215587258s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................X.........X.........] duration: 6.430781881s FAIL (2025-10-23 14:24:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:18:06.304157422 (720h46m51.766249905s) [upload=634034]: Starting upload Oct 23 14:18:06.304347725 (720h46m51.766440218s) [upload=634034]: Done starting upload Oct 23 14:18:06.304632076 (720h46m51.766724559s) [upload=634034]: Uploading segments... Oct 23 14:18:06.304690778 (720h46m51.766783261s) [upload=634034]: Got next segment. Inline: false Oct 23 14:18:06.304730654 (720h46m51.766823147s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:18:06.304737897 (720h46m51.766830371s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:18:06.560110142 (720h46m52.022202615s) [seg_pos={0 0}, upload=634034]: Begin upload segment... Oct 23 14:18:06.560170466 (720h46m52.022262949s) [seg_pos={0 0}, upload=634034]: Done begin upload segment. Oct 23 14:18:06.560196877 (720h46m52.02228936s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.560384274 (720h46m52.022476767s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561087734 (720h46m52.023180217s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561246465 (720h46m52.023338948s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561388856 (720h46m52.02348134s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561682246 (720h46m52.023774729s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561840206 (720h46m52.023932699s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562036520 (720h46m52.024129003s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562369515 (720h46m52.024461998s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562402808 (720h46m52.024495301s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562804713 (720h46m52.024897196s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562996849 (720h46m52.025089332s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563336766 (720h46m52.025429249s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563574799 (720h46m52.025667282s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564175863 (720h46m52.026268346s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564848324 (720h46m52.026940798s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:08.284021691 (720h46m53.746114164s) [upload=634034]: Next returned nil segment Oct 23 14:18:08.284044505 (720h46m53.746136978s) [upload=634034]: Waiting for error group managing segments... Oct 23 14:18:08.566598218 (720h46m54.028690691s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.594453381 (720h46m54.056545864s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.598953419 (720h46m54.061045892s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.602023124 (720h46m54.064115658s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625098415 (720h46m54.087190898s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625760095 (720h46m54.087852578s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.631004962 (720h46m54.093097445s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.635035678 (720h46m54.097128161s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.645903372 (720h46m54.107995845s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.711835221 (720h46m54.173927704s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.719309664 (720h46m54.181402147s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.758298877 (720h46m54.22039136s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819795110 (720h46m54.281887593s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819860023 (720h46m54.281952516s) [seg_pos={0 0}, upload=634034]: Segment reached optimal threshold of 13 pieces. Oct 23 14:18:08.820128314 (720h46m54.282220807s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection Oct 23 14:18:08.820140978 (720h46m54.282233461s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820198287 (720h46m54.28229077s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820209729 (720h46m54.282302222s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820445267 (720h46m54.28253775s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820509670 (720h46m54.282602153s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820531111 (720h46m54.282623594s) [seg_pos={0 0}, upload=634034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:18:08.820624719 (720h46m54.282717202s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:18:08.820630982 (720h46m54.282723465s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:18:09.121056837 (720h46m54.58314933s) [upload=634034]: Done uploading segments... Oct 23 14:18:09.121104197 (720h46m54.58319668s) [upload=634034]: Upload finished. err: <nil> Duration: 6.430825636s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.48767843s FAIL (2025-10-23 14:24:30 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.487710339s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 8.884453457s FAIL (2025-10-23 14:24:05 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 8.884493342s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 11.58244219s FAIL (2025-10-23 14:23:44 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 11.582475764s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X.........X........] duration: 6.164940687s FAIL (2025-10-23 14:23:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:18:06.304157422 (720h46m51.766249905s) [upload=634034]: Starting upload Oct 23 14:18:06.304347725 (720h46m51.766440218s) [upload=634034]: Done starting upload Oct 23 14:18:06.304632076 (720h46m51.766724559s) [upload=634034]: Uploading segments... Oct 23 14:18:06.304690778 (720h46m51.766783261s) [upload=634034]: Got next segment. Inline: false Oct 23 14:18:06.304730654 (720h46m51.766823147s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:18:06.304737897 (720h46m51.766830371s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:18:06.560110142 (720h46m52.022202615s) [seg_pos={0 0}, upload=634034]: Begin upload segment... Oct 23 14:18:06.560170466 (720h46m52.022262949s) [seg_pos={0 0}, upload=634034]: Done begin upload segment. Oct 23 14:18:06.560196877 (720h46m52.02228936s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.560384274 (720h46m52.022476767s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561087734 (720h46m52.023180217s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561246465 (720h46m52.023338948s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561388856 (720h46m52.02348134s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561682246 (720h46m52.023774729s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561840206 (720h46m52.023932699s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562036520 (720h46m52.024129003s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562369515 (720h46m52.024461998s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562402808 (720h46m52.024495301s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562804713 (720h46m52.024897196s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562996849 (720h46m52.025089332s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563336766 (720h46m52.025429249s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563574799 (720h46m52.025667282s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564175863 (720h46m52.026268346s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564848324 (720h46m52.026940798s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:08.284021691 (720h46m53.746114164s) [upload=634034]: Next returned nil segment Oct 23 14:18:08.284044505 (720h46m53.746136978s) [upload=634034]: Waiting for error group managing segments... Oct 23 14:18:08.566598218 (720h46m54.028690691s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.594453381 (720h46m54.056545864s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.598953419 (720h46m54.061045892s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.602023124 (720h46m54.064115658s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625098415 (720h46m54.087190898s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625760095 (720h46m54.087852578s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.631004962 (720h46m54.093097445s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.635035678 (720h46m54.097128161s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.645903372 (720h46m54.107995845s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.711835221 (720h46m54.173927704s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.719309664 (720h46m54.181402147s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.758298877 (720h46m54.22039136s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819795110 (720h46m54.281887593s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819860023 (720h46m54.281952516s) [seg_pos={0 0}, upload=634034]: Segment reached optimal threshold of 13 pieces. Oct 23 14:18:08.820128314 (720h46m54.282220807s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection Oct 23 14:18:08.820140978 (720h46m54.282233461s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820198287 (720h46m54.28229077s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820209729 (720h46m54.282302222s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820445267 (720h46m54.28253775s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820509670 (720h46m54.282602153s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820531111 (720h46m54.282623594s) [seg_pos={0 0}, upload=634034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:18:08.820624719 (720h46m54.282717202s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:18:08.820630982 (720h46m54.282723465s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:18:09.121056837 (720h46m54.58314933s) [upload=634034]: Done uploading segments... Oct 23 14:18:09.121104197 (720h46m54.58319668s) [upload=634034]: Upload finished. err: <nil> Duration: 6.16497331s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.544491943s FAIL (2025-10-23 14:23:24 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.544511661s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 7.317344266s FAIL (2025-10-23 14:23:19 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 7.317380725s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 9.680340899s FAIL (2025-10-23 14:22:36 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 9.680379842s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X.........X.......] duration: 9.061440448s FAIL (2025-10-23 14:22:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:18:06.304157422 (720h46m51.766249905s) [upload=634034]: Starting upload Oct 23 14:18:06.304347725 (720h46m51.766440218s) [upload=634034]: Done starting upload Oct 23 14:18:06.304632076 (720h46m51.766724559s) [upload=634034]: Uploading segments... Oct 23 14:18:06.304690778 (720h46m51.766783261s) [upload=634034]: Got next segment. Inline: false Oct 23 14:18:06.304730654 (720h46m51.766823147s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:18:06.304737897 (720h46m51.766830371s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:18:06.560110142 (720h46m52.022202615s) [seg_pos={0 0}, upload=634034]: Begin upload segment... Oct 23 14:18:06.560170466 (720h46m52.022262949s) [seg_pos={0 0}, upload=634034]: Done begin upload segment. Oct 23 14:18:06.560196877 (720h46m52.02228936s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.560384274 (720h46m52.022476767s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561087734 (720h46m52.023180217s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561246465 (720h46m52.023338948s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561388856 (720h46m52.02348134s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561682246 (720h46m52.023774729s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561840206 (720h46m52.023932699s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562036520 (720h46m52.024129003s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562369515 (720h46m52.024461998s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562402808 (720h46m52.024495301s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562804713 (720h46m52.024897196s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562996849 (720h46m52.025089332s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563336766 (720h46m52.025429249s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563574799 (720h46m52.025667282s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564175863 (720h46m52.026268346s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564848324 (720h46m52.026940798s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:08.284021691 (720h46m53.746114164s) [upload=634034]: Next returned nil segment Oct 23 14:18:08.284044505 (720h46m53.746136978s) [upload=634034]: Waiting for error group managing segments... Oct 23 14:18:08.566598218 (720h46m54.028690691s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.594453381 (720h46m54.056545864s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.598953419 (720h46m54.061045892s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.602023124 (720h46m54.064115658s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625098415 (720h46m54.087190898s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625760095 (720h46m54.087852578s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.631004962 (720h46m54.093097445s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.635035678 (720h46m54.097128161s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.645903372 (720h46m54.107995845s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.711835221 (720h46m54.173927704s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.719309664 (720h46m54.181402147s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.758298877 (720h46m54.22039136s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819795110 (720h46m54.281887593s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819860023 (720h46m54.281952516s) [seg_pos={0 0}, upload=634034]: Segment reached optimal threshold of 13 pieces. Oct 23 14:18:08.820128314 (720h46m54.282220807s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection Oct 23 14:18:08.820140978 (720h46m54.282233461s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820198287 (720h46m54.28229077s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820209729 (720h46m54.282302222s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820445267 (720h46m54.28253775s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820509670 (720h46m54.282602153s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820531111 (720h46m54.282623594s) [seg_pos={0 0}, upload=634034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:18:08.820624719 (720h46m54.282717202s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:18:08.820630982 (720h46m54.282723465s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:18:09.121056837 (720h46m54.58314933s) [upload=634034]: Done uploading segments... Oct 23 14:18:09.121104197 (720h46m54.58319668s) [upload=634034]: Upload finished. err: <nil> Duration: 9.061475296s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.504277036s FAIL (2025-10-23 14:22:33 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 2.504310279s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 7.735461306s FAIL (2025-10-23 14:22:06 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 7.735493538s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.694148982s FAIL (2025-10-23 14:22:05 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 2.694189037s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X.........X......] duration: 7.486910581s FAIL (2025-10-23 14:21:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:18:06.304157422 (720h46m51.766249905s) [upload=634034]: Starting upload Oct 23 14:18:06.304347725 (720h46m51.766440218s) [upload=634034]: Done starting upload Oct 23 14:18:06.304632076 (720h46m51.766724559s) [upload=634034]: Uploading segments... Oct 23 14:18:06.304690778 (720h46m51.766783261s) [upload=634034]: Got next segment. Inline: false Oct 23 14:18:06.304730654 (720h46m51.766823147s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:18:06.304737897 (720h46m51.766830371s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:18:06.560110142 (720h46m52.022202615s) [seg_pos={0 0}, upload=634034]: Begin upload segment... Oct 23 14:18:06.560170466 (720h46m52.022262949s) [seg_pos={0 0}, upload=634034]: Done begin upload segment. Oct 23 14:18:06.560196877 (720h46m52.02228936s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.560384274 (720h46m52.022476767s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561087734 (720h46m52.023180217s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561246465 (720h46m52.023338948s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561388856 (720h46m52.02348134s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561682246 (720h46m52.023774729s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561840206 (720h46m52.023932699s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562036520 (720h46m52.024129003s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562369515 (720h46m52.024461998s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562402808 (720h46m52.024495301s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562804713 (720h46m52.024897196s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562996849 (720h46m52.025089332s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563336766 (720h46m52.025429249s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563574799 (720h46m52.025667282s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564175863 (720h46m52.026268346s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564848324 (720h46m52.026940798s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:08.284021691 (720h46m53.746114164s) [upload=634034]: Next returned nil segment Oct 23 14:18:08.284044505 (720h46m53.746136978s) [upload=634034]: Waiting for error group managing segments... Oct 23 14:18:08.566598218 (720h46m54.028690691s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.594453381 (720h46m54.056545864s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.598953419 (720h46m54.061045892s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.602023124 (720h46m54.064115658s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625098415 (720h46m54.087190898s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625760095 (720h46m54.087852578s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.631004962 (720h46m54.093097445s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.635035678 (720h46m54.097128161s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.645903372 (720h46m54.107995845s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.711835221 (720h46m54.173927704s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.719309664 (720h46m54.181402147s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.758298877 (720h46m54.22039136s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819795110 (720h46m54.281887593s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819860023 (720h46m54.281952516s) [seg_pos={0 0}, upload=634034]: Segment reached optimal threshold of 13 pieces. Oct 23 14:18:08.820128314 (720h46m54.282220807s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection Oct 23 14:18:08.820140978 (720h46m54.282233461s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820198287 (720h46m54.28229077s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820209729 (720h46m54.282302222s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820445267 (720h46m54.28253775s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820509670 (720h46m54.282602153s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820531111 (720h46m54.282623594s) [seg_pos={0 0}, upload=634034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:18:08.820624719 (720h46m54.282717202s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:18:08.820630982 (720h46m54.282723465s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:18:09.121056837 (720h46m54.58314933s) [upload=634034]: Done uploading segments... Oct 23 14:18:09.121104197 (720h46m54.58319668s) [upload=634034]: Upload finished. err: <nil> Duration: 7.486937653s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 7.555418811s FAIL (2025-10-23 14:21:14 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 7.55547058s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................X.........X.....] duration: 8.237154224s FAIL (2025-10-23 14:21:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:18:06.304157422 (720h46m51.766249905s) [upload=634034]: Starting upload Oct 23 14:18:06.304347725 (720h46m51.766440218s) [upload=634034]: Done starting upload Oct 23 14:18:06.304632076 (720h46m51.766724559s) [upload=634034]: Uploading segments... Oct 23 14:18:06.304690778 (720h46m51.766783261s) [upload=634034]: Got next segment. Inline: false Oct 23 14:18:06.304730654 (720h46m51.766823147s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:18:06.304737897 (720h46m51.766830371s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:18:06.560110142 (720h46m52.022202615s) [seg_pos={0 0}, upload=634034]: Begin upload segment... Oct 23 14:18:06.560170466 (720h46m52.022262949s) [seg_pos={0 0}, upload=634034]: Done begin upload segment. Oct 23 14:18:06.560196877 (720h46m52.02228936s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.560384274 (720h46m52.022476767s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561087734 (720h46m52.023180217s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561246465 (720h46m52.023338948s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561388856 (720h46m52.02348134s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561682246 (720h46m52.023774729s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561840206 (720h46m52.023932699s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562036520 (720h46m52.024129003s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562369515 (720h46m52.024461998s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562402808 (720h46m52.024495301s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562804713 (720h46m52.024897196s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562996849 (720h46m52.025089332s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563336766 (720h46m52.025429249s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563574799 (720h46m52.025667282s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564175863 (720h46m52.026268346s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564848324 (720h46m52.026940798s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:08.284021691 (720h46m53.746114164s) [upload=634034]: Next returned nil segment Oct 23 14:18:08.284044505 (720h46m53.746136978s) [upload=634034]: Waiting for error group managing segments... Oct 23 14:18:08.566598218 (720h46m54.028690691s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.594453381 (720h46m54.056545864s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.598953419 (720h46m54.061045892s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.602023124 (720h46m54.064115658s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625098415 (720h46m54.087190898s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625760095 (720h46m54.087852578s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.631004962 (720h46m54.093097445s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.635035678 (720h46m54.097128161s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.645903372 (720h46m54.107995845s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.711835221 (720h46m54.173927704s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.719309664 (720h46m54.181402147s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.758298877 (720h46m54.22039136s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819795110 (720h46m54.281887593s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819860023 (720h46m54.281952516s) [seg_pos={0 0}, upload=634034]: Segment reached optimal threshold of 13 pieces. Oct 23 14:18:08.820128314 (720h46m54.282220807s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection Oct 23 14:18:08.820140978 (720h46m54.282233461s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820198287 (720h46m54.28229077s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820209729 (720h46m54.282302222s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820445267 (720h46m54.28253775s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820509670 (720h46m54.282602153s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820531111 (720h46m54.282623594s) [seg_pos={0 0}, upload=634034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:18:08.820624719 (720h46m54.282717202s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:18:08.820630982 (720h46m54.282723465s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:18:09.121056837 (720h46m54.58314933s) [upload=634034]: Done uploading segments... Oct 23 14:18:09.121104197 (720h46m54.58319668s) [upload=634034]: Upload finished. err: <nil> Duration: 8.237180472s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.26314664s FAIL (2025-10-23 14:21:02 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 2.263179392s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X.........X....] duration: 9.424724461s FAIL (2025-10-23 14:20:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:18:06.304157422 (720h46m51.766249905s) [upload=634034]: Starting upload Oct 23 14:18:06.304347725 (720h46m51.766440218s) [upload=634034]: Done starting upload Oct 23 14:18:06.304632076 (720h46m51.766724559s) [upload=634034]: Uploading segments... Oct 23 14:18:06.304690778 (720h46m51.766783261s) [upload=634034]: Got next segment. Inline: false Oct 23 14:18:06.304730654 (720h46m51.766823147s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:18:06.304737897 (720h46m51.766830371s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:18:06.560110142 (720h46m52.022202615s) [seg_pos={0 0}, upload=634034]: Begin upload segment... Oct 23 14:18:06.560170466 (720h46m52.022262949s) [seg_pos={0 0}, upload=634034]: Done begin upload segment. Oct 23 14:18:06.560196877 (720h46m52.02228936s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.560384274 (720h46m52.022476767s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561087734 (720h46m52.023180217s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561246465 (720h46m52.023338948s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561388856 (720h46m52.02348134s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561682246 (720h46m52.023774729s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561840206 (720h46m52.023932699s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562036520 (720h46m52.024129003s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562369515 (720h46m52.024461998s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562402808 (720h46m52.024495301s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562804713 (720h46m52.024897196s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562996849 (720h46m52.025089332s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563336766 (720h46m52.025429249s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563574799 (720h46m52.025667282s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564175863 (720h46m52.026268346s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564848324 (720h46m52.026940798s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:08.284021691 (720h46m53.746114164s) [upload=634034]: Next returned nil segment Oct 23 14:18:08.284044505 (720h46m53.746136978s) [upload=634034]: Waiting for error group managing segments... Oct 23 14:18:08.566598218 (720h46m54.028690691s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.594453381 (720h46m54.056545864s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.598953419 (720h46m54.061045892s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.602023124 (720h46m54.064115658s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625098415 (720h46m54.087190898s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625760095 (720h46m54.087852578s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.631004962 (720h46m54.093097445s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.635035678 (720h46m54.097128161s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.645903372 (720h46m54.107995845s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.711835221 (720h46m54.173927704s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.719309664 (720h46m54.181402147s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.758298877 (720h46m54.22039136s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819795110 (720h46m54.281887593s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819860023 (720h46m54.281952516s) [seg_pos={0 0}, upload=634034]: Segment reached optimal threshold of 13 pieces. Oct 23 14:18:08.820128314 (720h46m54.282220807s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection Oct 23 14:18:08.820140978 (720h46m54.282233461s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820198287 (720h46m54.28229077s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820209729 (720h46m54.282302222s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820445267 (720h46m54.28253775s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820509670 (720h46m54.282602153s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820531111 (720h46m54.282623594s) [seg_pos={0 0}, upload=634034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:18:08.820624719 (720h46m54.282717202s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:18:08.820630982 (720h46m54.282723465s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:18:09.121056837 (720h46m54.58314933s) [upload=634034]: Done uploading segments... Oct 23 14:18:09.121104197 (720h46m54.58319668s) [upload=634034]: Upload finished. err: <nil> Duration: 9.424761912s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 13.351236579s FAIL (2025-10-23 14:20:40 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 13.351264973s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X.........X...] duration: 10.313837622s FAIL (2025-10-23 14:20:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:18:06.304157422 (720h46m51.766249905s) [upload=634034]: Starting upload Oct 23 14:18:06.304347725 (720h46m51.766440218s) [upload=634034]: Done starting upload Oct 23 14:18:06.304632076 (720h46m51.766724559s) [upload=634034]: Uploading segments... Oct 23 14:18:06.304690778 (720h46m51.766783261s) [upload=634034]: Got next segment. Inline: false Oct 23 14:18:06.304730654 (720h46m51.766823147s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:18:06.304737897 (720h46m51.766830371s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:18:06.560110142 (720h46m52.022202615s) [seg_pos={0 0}, upload=634034]: Begin upload segment... Oct 23 14:18:06.560170466 (720h46m52.022262949s) [seg_pos={0 0}, upload=634034]: Done begin upload segment. Oct 23 14:18:06.560196877 (720h46m52.02228936s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.560384274 (720h46m52.022476767s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561087734 (720h46m52.023180217s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561246465 (720h46m52.023338948s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561388856 (720h46m52.02348134s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561682246 (720h46m52.023774729s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561840206 (720h46m52.023932699s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562036520 (720h46m52.024129003s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562369515 (720h46m52.024461998s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562402808 (720h46m52.024495301s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562804713 (720h46m52.024897196s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562996849 (720h46m52.025089332s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563336766 (720h46m52.025429249s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563574799 (720h46m52.025667282s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564175863 (720h46m52.026268346s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564848324 (720h46m52.026940798s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:08.284021691 (720h46m53.746114164s) [upload=634034]: Next returned nil segment Oct 23 14:18:08.284044505 (720h46m53.746136978s) [upload=634034]: Waiting for error group managing segments... Oct 23 14:18:08.566598218 (720h46m54.028690691s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.594453381 (720h46m54.056545864s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.598953419 (720h46m54.061045892s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.602023124 (720h46m54.064115658s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625098415 (720h46m54.087190898s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625760095 (720h46m54.087852578s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.631004962 (720h46m54.093097445s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.635035678 (720h46m54.097128161s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.645903372 (720h46m54.107995845s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.711835221 (720h46m54.173927704s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.719309664 (720h46m54.181402147s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.758298877 (720h46m54.22039136s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819795110 (720h46m54.281887593s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819860023 (720h46m54.281952516s) [seg_pos={0 0}, upload=634034]: Segment reached optimal threshold of 13 pieces. Oct 23 14:18:08.820128314 (720h46m54.282220807s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection Oct 23 14:18:08.820140978 (720h46m54.282233461s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820198287 (720h46m54.28229077s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820209729 (720h46m54.282302222s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820445267 (720h46m54.28253775s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820509670 (720h46m54.282602153s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820531111 (720h46m54.282623594s) [seg_pos={0 0}, upload=634034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:18:08.820624719 (720h46m54.282717202s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:18:08.820630982 (720h46m54.282723465s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:18:09.121056837 (720h46m54.58314933s) [upload=634034]: Done uploading segments... Oct 23 14:18:09.121104197 (720h46m54.58319668s) [upload=634034]: Upload finished. err: <nil> Duration: 10.313871887s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 6.3840215s FAIL (2025-10-23 14:20:06 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.384059463s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.132629889s FAIL (2025-10-23 14:19:53 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 2.132661408s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X.........X..] duration: 5.904783561s FAIL (2025-10-23 14:19:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:18:06.304157422 (720h46m51.766249905s) [upload=634034]: Starting upload Oct 23 14:18:06.304347725 (720h46m51.766440218s) [upload=634034]: Done starting upload Oct 23 14:18:06.304632076 (720h46m51.766724559s) [upload=634034]: Uploading segments... Oct 23 14:18:06.304690778 (720h46m51.766783261s) [upload=634034]: Got next segment. Inline: false Oct 23 14:18:06.304730654 (720h46m51.766823147s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:18:06.304737897 (720h46m51.766830371s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:18:06.560110142 (720h46m52.022202615s) [seg_pos={0 0}, upload=634034]: Begin upload segment... Oct 23 14:18:06.560170466 (720h46m52.022262949s) [seg_pos={0 0}, upload=634034]: Done begin upload segment. Oct 23 14:18:06.560196877 (720h46m52.02228936s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.560384274 (720h46m52.022476767s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561087734 (720h46m52.023180217s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561246465 (720h46m52.023338948s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561388856 (720h46m52.02348134s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561682246 (720h46m52.023774729s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561840206 (720h46m52.023932699s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562036520 (720h46m52.024129003s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562369515 (720h46m52.024461998s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562402808 (720h46m52.024495301s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562804713 (720h46m52.024897196s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562996849 (720h46m52.025089332s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563336766 (720h46m52.025429249s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563574799 (720h46m52.025667282s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564175863 (720h46m52.026268346s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564848324 (720h46m52.026940798s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:08.284021691 (720h46m53.746114164s) [upload=634034]: Next returned nil segment Oct 23 14:18:08.284044505 (720h46m53.746136978s) [upload=634034]: Waiting for error group managing segments... Oct 23 14:18:08.566598218 (720h46m54.028690691s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.594453381 (720h46m54.056545864s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.598953419 (720h46m54.061045892s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.602023124 (720h46m54.064115658s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625098415 (720h46m54.087190898s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625760095 (720h46m54.087852578s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.631004962 (720h46m54.093097445s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.635035678 (720h46m54.097128161s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.645903372 (720h46m54.107995845s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.711835221 (720h46m54.173927704s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.719309664 (720h46m54.181402147s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.758298877 (720h46m54.22039136s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819795110 (720h46m54.281887593s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819860023 (720h46m54.281952516s) [seg_pos={0 0}, upload=634034]: Segment reached optimal threshold of 13 pieces. Oct 23 14:18:08.820128314 (720h46m54.282220807s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection Oct 23 14:18:08.820140978 (720h46m54.282233461s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820198287 (720h46m54.28229077s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820209729 (720h46m54.282302222s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820445267 (720h46m54.28253775s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820509670 (720h46m54.282602153s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820531111 (720h46m54.282623594s) [seg_pos={0 0}, upload=634034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:18:08.820624719 (720h46m54.282717202s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:18:08.820630982 (720h46m54.282723465s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:18:09.121056837 (720h46m54.58314933s) [upload=634034]: Done uploading segments... Oct 23 14:18:09.121104197 (720h46m54.58319668s) [upload=634034]: Upload finished. err: <nil> Duration: 5.904814s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.061407153s FAIL (2025-10-23 14:19:25 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 2.061439937s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.261740216s FAIL (2025-10-23 14:19:08 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.261768831s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X.........X.] duration: 5.852867953s FAIL (2025-10-23 14:19:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:18:06.304157422 (720h46m51.766249905s) [upload=634034]: Starting upload Oct 23 14:18:06.304347725 (720h46m51.766440218s) [upload=634034]: Done starting upload Oct 23 14:18:06.304632076 (720h46m51.766724559s) [upload=634034]: Uploading segments... Oct 23 14:18:06.304690778 (720h46m51.766783261s) [upload=634034]: Got next segment. Inline: false Oct 23 14:18:06.304730654 (720h46m51.766823147s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:18:06.304737897 (720h46m51.766830371s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:18:06.560110142 (720h46m52.022202615s) [seg_pos={0 0}, upload=634034]: Begin upload segment... Oct 23 14:18:06.560170466 (720h46m52.022262949s) [seg_pos={0 0}, upload=634034]: Done begin upload segment. Oct 23 14:18:06.560196877 (720h46m52.02228936s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.560384274 (720h46m52.022476767s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561087734 (720h46m52.023180217s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561246465 (720h46m52.023338948s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561388856 (720h46m52.02348134s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561682246 (720h46m52.023774729s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561840206 (720h46m52.023932699s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562036520 (720h46m52.024129003s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562369515 (720h46m52.024461998s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562402808 (720h46m52.024495301s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562804713 (720h46m52.024897196s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562996849 (720h46m52.025089332s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563336766 (720h46m52.025429249s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563574799 (720h46m52.025667282s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564175863 (720h46m52.026268346s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564848324 (720h46m52.026940798s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:08.284021691 (720h46m53.746114164s) [upload=634034]: Next returned nil segment Oct 23 14:18:08.284044505 (720h46m53.746136978s) [upload=634034]: Waiting for error group managing segments... Oct 23 14:18:08.566598218 (720h46m54.028690691s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.594453381 (720h46m54.056545864s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.598953419 (720h46m54.061045892s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.602023124 (720h46m54.064115658s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625098415 (720h46m54.087190898s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625760095 (720h46m54.087852578s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.631004962 (720h46m54.093097445s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.635035678 (720h46m54.097128161s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.645903372 (720h46m54.107995845s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.711835221 (720h46m54.173927704s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.719309664 (720h46m54.181402147s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.758298877 (720h46m54.22039136s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819795110 (720h46m54.281887593s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819860023 (720h46m54.281952516s) [seg_pos={0 0}, upload=634034]: Segment reached optimal threshold of 13 pieces. Oct 23 14:18:08.820128314 (720h46m54.282220807s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection Oct 23 14:18:08.820140978 (720h46m54.282233461s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820198287 (720h46m54.28229077s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820209729 (720h46m54.282302222s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820445267 (720h46m54.28253775s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820509670 (720h46m54.282602153s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820531111 (720h46m54.282623594s) [seg_pos={0 0}, upload=634034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:18:08.820624719 (720h46m54.282717202s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:18:08.820630982 (720h46m54.282723465s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:18:09.121056837 (720h46m54.58314933s) [upload=634034]: Done uploading segments... Oct 23 14:18:09.121104197 (720h46m54.58319668s) [upload=634034]: Upload finished. err: <nil> Duration: 5.852904792s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.047187808s FAIL (2025-10-23 14:18:57 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 2.047229708s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 7.465272729s FAIL (2025-10-23 14:18:53 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 7.465307355s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X.........X] duration: 30.002768915s FAIL (2025-10-23 14:18:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:18:06.304157422 (720h46m51.766249905s) [upload=634034]: Starting upload Oct 23 14:18:06.304347725 (720h46m51.766440218s) [upload=634034]: Done starting upload Oct 23 14:18:06.304632076 (720h46m51.766724559s) [upload=634034]: Uploading segments... Oct 23 14:18:06.304690778 (720h46m51.766783261s) [upload=634034]: Got next segment. Inline: false Oct 23 14:18:06.304730654 (720h46m51.766823147s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:18:06.304737897 (720h46m51.766830371s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:18:06.560110142 (720h46m52.022202615s) [seg_pos={0 0}, upload=634034]: Begin upload segment... Oct 23 14:18:06.560170466 (720h46m52.022262949s) [seg_pos={0 0}, upload=634034]: Done begin upload segment. Oct 23 14:18:06.560196877 (720h46m52.02228936s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.560384274 (720h46m52.022476767s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561087734 (720h46m52.023180217s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561246465 (720h46m52.023338948s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561388856 (720h46m52.02348134s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561682246 (720h46m52.023774729s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561840206 (720h46m52.023932699s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562036520 (720h46m52.024129003s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562369515 (720h46m52.024461998s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562402808 (720h46m52.024495301s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562804713 (720h46m52.024897196s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562996849 (720h46m52.025089332s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563336766 (720h46m52.025429249s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563574799 (720h46m52.025667282s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564175863 (720h46m52.026268346s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564848324 (720h46m52.026940798s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:08.284021691 (720h46m53.746114164s) [upload=634034]: Next returned nil segment Oct 23 14:18:08.284044505 (720h46m53.746136978s) [upload=634034]: Waiting for error group managing segments... Oct 23 14:18:08.566598218 (720h46m54.028690691s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.594453381 (720h46m54.056545864s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.598953419 (720h46m54.061045892s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.602023124 (720h46m54.064115658s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625098415 (720h46m54.087190898s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625760095 (720h46m54.087852578s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.631004962 (720h46m54.093097445s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.635035678 (720h46m54.097128161s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.645903372 (720h46m54.107995845s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.711835221 (720h46m54.173927704s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.719309664 (720h46m54.181402147s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.758298877 (720h46m54.22039136s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819795110 (720h46m54.281887593s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819860023 (720h46m54.281952516s) [seg_pos={0 0}, upload=634034]: Segment reached optimal threshold of 13 pieces. Oct 23 14:18:08.820128314 (720h46m54.282220807s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection Oct 23 14:18:08.820140978 (720h46m54.282233461s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820198287 (720h46m54.28229077s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820209729 (720h46m54.282302222s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820445267 (720h46m54.28253775s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820509670 (720h46m54.282602153s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820531111 (720h46m54.282623594s) [seg_pos={0 0}, upload=634034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:18:08.820624719 (720h46m54.282717202s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:18:08.820630982 (720h46m54.282723465s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:18:09.121056837 (720h46m54.58314933s) [upload=634034]: Done uploading segments... Oct 23 14:18:09.121104197 (720h46m54.58319668s) [upload=634034]: Upload finished. err: <nil> Duration: 30.002793241s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.120024642s FAIL (2025-10-23 14:18:26 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.120062845s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 7.155095152s FAIL (2025-10-23 14:18:15 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 7.155131401s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 6.067308061s FAIL (2025-10-23 14:17:52 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.067329532s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.98497952s FAIL (2025-10-23 14:17:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:12:11.907236120 (720h40m57.369328593s) [upload=633955]: Starting upload Oct 23 14:12:11.907307306 (720h40m57.369399789s) [upload=633955]: Done starting upload Oct 23 14:12:11.907391667 (720h40m57.36948415s) [upload=633955]: Uploading segments... Oct 23 14:12:11.907493661 (720h40m57.369586154s) [upload=633955]: Got next segment. Inline: false Oct 23 14:12:11.907561601 (720h40m57.369654084s) [upload=633955]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:12:11.907568914 (720h40m57.369661417s) [upload=633955]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:12:12.459592256 (720h40m57.921684739s) [seg_pos={0 0}, upload=633955]: Begin upload segment... Oct 23 14:12:12.459664634 (720h40m57.921757117s) [seg_pos={0 0}, upload=633955]: Done begin upload segment. Oct 23 14:12:12.459686255 (720h40m57.921778728s) [noise=true , address=145.79.227.3:4150 , piece_id=LDZ4NVA7GSORO53QARUCDMJIOR5WGVIZJEQ5KAZRTDF4Z5764DTA, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.459730529 (720h40m57.921823012s) [noise=true , address=145.79.227.16:4909 , piece_id=IYIYJMIHNE62CBOEQF4N6PGGX3OICHC2OGPBQ7IDULBD2SFTHG6Q, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.459866769 (720h40m57.921959252s) [noise=true , address=145.79.227.19:4077 , piece_id=QQCCQ2OHYBHEFER257M7ZTAE7FTOQKZM6EFKZOAJZMTUI776XRVQ, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.460667133 (720h40m57.922759616s) [noise=true , address=77.90.31.12:4592 , piece_id=5TVQIJTVNQHQZAS5ZD2NUH5TZINEXPYP3QYO2BNMITKN7RBIOOVA, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.461059931 (720h40m57.923152414s) [noise=true , address=77.90.31.13:4626 , piece_id=3UWMJRRQU5G6LQGTG2RABD3UE46SMUBPQIQUDYIJDCJ224J5AKSQ, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.461441607 (720h40m57.923534091s) [noise=true , address=77.90.31.11:4553 , piece_id=WBW4C6GESNPKZLLPRAFHDG2SOUAJSYEYEYRQV4TTBXI4NFS6TNQQ, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.462275826 (720h40m57.924368309s) [noise=true , address=145.79.227.14:4840 , piece_id=SYZKKMI2NKG3I7C4YYRC25QVLO5CMOASWYOE75DAY45YPDSY7HRQ, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.462659697 (720h40m57.92475218s) [noise=true , address=145.79.227.26:4654 , piece_id=XDFYPG5ILF2UFYK76M455LWP5TWYIMAYWINUXYUHOV6ETL2DGG4Q, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.463142947 (720h40m57.92523542s) [noise=true , address=145.79.227.23:4545 , piece_id=TQL2O7TQ6AX3ADEVMNSMPRLTB4LRY7IHNMDGT5RTKGIBHPYBGV6Q, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.463514605 (720h40m57.925607078s) [noise=true , address=77.90.31.16:4735 , piece_id=5ZOPVHHAEOAIRVH2XJ3PHZUFP4YXSAU5NB5FT5RJLJNVY3D6KUDQ, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.464034425 (720h40m57.926126898s) [noise=true , address=145.79.227.20:4439 , piece_id=JHD5XPCBGEMDP67IFSKMZJQI7FREAFFH7N3F42MU7JYHY32T6R4Q, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.464441741 (720h40m57.926534214s) [noise=true , address=145.79.227.7:4294 , piece_id=32JC2H25VALZ3SQFD4XUK2HZSR437WUNIRE7TTBAU2NEKVCE5L4Q, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.464628176 (720h40m57.926720659s) [noise=true , address=77.90.31.10:4517 , piece_id=XJFYCOQDGEJN2YW45GSRURMYRBIVJDXVN4JDBZHDZZBBDK5W2AAQ, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.465067192 (720h40m57.927159675s) [noise=true , address=145.79.227.14:4839 , piece_id=UR62ATUIMJJQ7UZC5T7IWYA5Z35STVWECHG7HY6OJGYHYHJ467JQ, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.468800320 (720h40m57.930892803s) [noise=true , address=145.79.227.11:4730 , piece_id=GJUKYXUPMETZCTIXQPQDPJJL3YSIPEOIO23KMBA33IQUZTV4GGEA, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.470230053 (720h40m57.932322536s) [noise=true , address=45.91.116.6:4975 , piece_id=SKM7JOZ3W5RLS25ZOA6222CG7DZ3M6I2W2N2353YFHYNPTGM6L3Q, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:14.111778086 (720h40m59.573870569s) [upload=633955]: Next returned nil segment Oct 23 14:12:14.111799357 (720h40m59.57389184s) [upload=633955]: Waiting for error group managing segments... Oct 23 14:12:14.498715648 (720h40m59.960808132s) [noise=true , address=145.79.227.3:4150 , piece_id=LDZ4NVA7GSORO53QARUCDMJIOR5WGVIZJEQ5KAZRTDF4Z5764DTA, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.510890272 (720h40m59.972982755s) [noise=true , address=145.79.227.16:4909 , piece_id=IYIYJMIHNE62CBOEQF4N6PGGX3OICHC2OGPBQ7IDULBD2SFTHG6Q, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.511458444 (720h40m59.973550927s) [noise=true , address=145.79.227.7:4294 , piece_id=32JC2H25VALZ3SQFD4XUK2HZSR437WUNIRE7TTBAU2NEKVCE5L4Q, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.526524743 (720h40m59.988617226s) [noise=true , address=145.79.227.14:4840 , piece_id=SYZKKMI2NKG3I7C4YYRC25QVLO5CMOASWYOE75DAY45YPDSY7HRQ, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.530430180 (720h40m59.992522663s) [noise=true , address=145.79.227.20:4439 , piece_id=JHD5XPCBGEMDP67IFSKMZJQI7FREAFFH7N3F42MU7JYHY32T6R4Q, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.588451766 (720h41m0.050544249s) [noise=true , address=145.79.227.14:4839 , piece_id=UR62ATUIMJJQ7UZC5T7IWYA5Z35STVWECHG7HY6OJGYHYHJ467JQ, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.588973710 (720h41m0.051066193s) [noise=true , address=145.79.227.19:4077 , piece_id=QQCCQ2OHYBHEFER257M7ZTAE7FTOQKZM6EFKZOAJZMTUI776XRVQ, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.603144364 (720h41m0.065236847s) [noise=true , address=145.79.227.26:4654 , piece_id=XDFYPG5ILF2UFYK76M455LWP5TWYIMAYWINUXYUHOV6ETL2DGG4Q, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.615118234 (720h41m0.077210717s) [noise=true , address=145.79.227.11:4730 , piece_id=GJUKYXUPMETZCTIXQPQDPJJL3YSIPEOIO23KMBA33IQUZTV4GGEA, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.677591017 (720h41m0.1396835s) [noise=true , address=77.90.31.12:4592 , piece_id=5TVQIJTVNQHQZAS5ZD2NUH5TZINEXPYP3QYO2BNMITKN7RBIOOVA, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.748693641 (720h41m0.210786124s) [noise=true , address=45.91.116.6:4975 , piece_id=SKM7JOZ3W5RLS25ZOA6222CG7DZ3M6I2W2N2353YFHYNPTGM6L3Q, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.800596134 (720h41m0.262688617s) [noise=true , address=77.90.31.11:4553 , piece_id=WBW4C6GESNPKZLLPRAFHDG2SOUAJSYEYEYRQV4TTBXI4NFS6TNQQ, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.809229134 (720h41m0.271321617s) [noise=true , address=77.90.31.16:4735 , piece_id=5ZOPVHHAEOAIRVH2XJ3PHZUFP4YXSAU5NB5FT5RJLJNVY3D6KUDQ, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.809288828 (720h41m0.271381311s) [seg_pos={0 0}, upload=633955]: Segment reached optimal threshold of 13 pieces. Oct 23 14:12:14.809633875 (720h41m0.271726358s) [noise=true , address=77.90.31.13:4626 , piece_id=3UWMJRRQU5G6LQGTG2RABD3UE46SMUBPQIQUDYIJDCJ224J5AKSQ, seg_pos={0 0}, upload=633955]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Oct 23 14:12:14.809661617 (720h41m0.27175409s) [noise=true , address=77.90.31.13:4626 , piece_id=3UWMJRRQU5G6LQGTG2RABD3UE46SMUBPQIQUDYIJDCJ224J5AKSQ, seg_pos={0 0}, upload=633955]: Long tail cancellation due to optimal threshold Oct 23 14:12:14.809823636 (720h41m0.271916109s) [noise=true , address=145.79.227.23:4545 , piece_id=TQL2O7TQ6AX3ADEVMNSMPRLTB4LRY7IHNMDGT5RTKGIBHPYBGV6Q, seg_pos={0 0}, upload=633955]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra): protocol: write tcp 188.166.225.170:36684->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:36684->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36684->145.79.227.23:4545: use of closed network connection Oct 23 14:12:14.809834356 (720h41m0.271926829s) [noise=true , address=145.79.227.23:4545 , piece_id=TQL2O7TQ6AX3ADEVMNSMPRLTB4LRY7IHNMDGT5RTKGIBHPYBGV6Q, seg_pos={0 0}, upload=633955]: Long tail cancellation due to optimal threshold Oct 23 14:12:14.809966377 (720h41m0.27205886s) [noise=true , address=77.90.31.10:4517 , piece_id=XJFYCOQDGEJN2YW45GSRURMYRBIVJDXVN4JDBZHDZZBBDK5W2AAQ, seg_pos={0 0}, upload=633955]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Oct 23 14:12:14.809979944 (720h41m0.272072427s) [noise=true , address=77.90.31.10:4517 , piece_id=XJFYCOQDGEJN2YW45GSRURMYRBIVJDXVN4JDBZHDZZBBDK5W2AAQ, seg_pos={0 0}, upload=633955]: Long tail cancellation due to optimal threshold Oct 23 14:12:14.810045387 (720h41m0.272137871s) [seg_pos={0 0}, upload=633955]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:12:14.810271368 (720h41m0.272363861s) [upload=633955]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:12:14.810281527 (720h41m0.27237401s) [upload=633955]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:12:15.146250265 (720h41m0.608342758s) [upload=633955]: Done uploading segments... Oct 23 14:12:15.146275132 (720h41m0.608367615s) [upload=633955]: Upload finished. err: <nil> Duration: 5.985014839s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.119359521s FAIL (2025-10-23 14:17:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:12:11.907236120 (720h40m57.369328593s) [upload=633955]: Starting upload Oct 23 14:12:11.907307306 (720h40m57.369399789s) [upload=633955]: Done starting upload Oct 23 14:12:11.907391667 (720h40m57.36948415s) [upload=633955]: Uploading segments... Oct 23 14:12:11.907493661 (720h40m57.369586154s) [upload=633955]: Got next segment. Inline: false Oct 23 14:12:11.907561601 (720h40m57.369654084s) [upload=633955]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:12:11.907568914 (720h40m57.369661417s) [upload=633955]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:12:12.459592256 (720h40m57.921684739s) [seg_pos={0 0}, upload=633955]: Begin upload segment... Oct 23 14:12:12.459664634 (720h40m57.921757117s) [seg_pos={0 0}, upload=633955]: Done begin upload segment. Oct 23 14:12:12.459686255 (720h40m57.921778728s) [noise=true , address=145.79.227.3:4150 , piece_id=LDZ4NVA7GSORO53QARUCDMJIOR5WGVIZJEQ5KAZRTDF4Z5764DTA, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.459730529 (720h40m57.921823012s) [noise=true , address=145.79.227.16:4909 , piece_id=IYIYJMIHNE62CBOEQF4N6PGGX3OICHC2OGPBQ7IDULBD2SFTHG6Q, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.459866769 (720h40m57.921959252s) [noise=true , address=145.79.227.19:4077 , piece_id=QQCCQ2OHYBHEFER257M7ZTAE7FTOQKZM6EFKZOAJZMTUI776XRVQ, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.460667133 (720h40m57.922759616s) [noise=true , address=77.90.31.12:4592 , piece_id=5TVQIJTVNQHQZAS5ZD2NUH5TZINEXPYP3QYO2BNMITKN7RBIOOVA, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.461059931 (720h40m57.923152414s) [noise=true , address=77.90.31.13:4626 , piece_id=3UWMJRRQU5G6LQGTG2RABD3UE46SMUBPQIQUDYIJDCJ224J5AKSQ, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.461441607 (720h40m57.923534091s) [noise=true , address=77.90.31.11:4553 , piece_id=WBW4C6GESNPKZLLPRAFHDG2SOUAJSYEYEYRQV4TTBXI4NFS6TNQQ, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.462275826 (720h40m57.924368309s) [noise=true , address=145.79.227.14:4840 , piece_id=SYZKKMI2NKG3I7C4YYRC25QVLO5CMOASWYOE75DAY45YPDSY7HRQ, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.462659697 (720h40m57.92475218s) [noise=true , address=145.79.227.26:4654 , piece_id=XDFYPG5ILF2UFYK76M455LWP5TWYIMAYWINUXYUHOV6ETL2DGG4Q, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.463142947 (720h40m57.92523542s) [noise=true , address=145.79.227.23:4545 , piece_id=TQL2O7TQ6AX3ADEVMNSMPRLTB4LRY7IHNMDGT5RTKGIBHPYBGV6Q, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.463514605 (720h40m57.925607078s) [noise=true , address=77.90.31.16:4735 , piece_id=5ZOPVHHAEOAIRVH2XJ3PHZUFP4YXSAU5NB5FT5RJLJNVY3D6KUDQ, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.464034425 (720h40m57.926126898s) [noise=true , address=145.79.227.20:4439 , piece_id=JHD5XPCBGEMDP67IFSKMZJQI7FREAFFH7N3F42MU7JYHY32T6R4Q, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.464441741 (720h40m57.926534214s) [noise=true , address=145.79.227.7:4294 , piece_id=32JC2H25VALZ3SQFD4XUK2HZSR437WUNIRE7TTBAU2NEKVCE5L4Q, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.464628176 (720h40m57.926720659s) [noise=true , address=77.90.31.10:4517 , piece_id=XJFYCOQDGEJN2YW45GSRURMYRBIVJDXVN4JDBZHDZZBBDK5W2AAQ, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.465067192 (720h40m57.927159675s) [noise=true , address=145.79.227.14:4839 , piece_id=UR62ATUIMJJQ7UZC5T7IWYA5Z35STVWECHG7HY6OJGYHYHJ467JQ, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.468800320 (720h40m57.930892803s) [noise=true , address=145.79.227.11:4730 , piece_id=GJUKYXUPMETZCTIXQPQDPJJL3YSIPEOIO23KMBA33IQUZTV4GGEA, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.470230053 (720h40m57.932322536s) [noise=true , address=45.91.116.6:4975 , piece_id=SKM7JOZ3W5RLS25ZOA6222CG7DZ3M6I2W2N2353YFHYNPTGM6L3Q, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:14.111778086 (720h40m59.573870569s) [upload=633955]: Next returned nil segment Oct 23 14:12:14.111799357 (720h40m59.57389184s) [upload=633955]: Waiting for error group managing segments... Oct 23 14:12:14.498715648 (720h40m59.960808132s) [noise=true , address=145.79.227.3:4150 , piece_id=LDZ4NVA7GSORO53QARUCDMJIOR5WGVIZJEQ5KAZRTDF4Z5764DTA, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.510890272 (720h40m59.972982755s) [noise=true , address=145.79.227.16:4909 , piece_id=IYIYJMIHNE62CBOEQF4N6PGGX3OICHC2OGPBQ7IDULBD2SFTHG6Q, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.511458444 (720h40m59.973550927s) [noise=true , address=145.79.227.7:4294 , piece_id=32JC2H25VALZ3SQFD4XUK2HZSR437WUNIRE7TTBAU2NEKVCE5L4Q, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.526524743 (720h40m59.988617226s) [noise=true , address=145.79.227.14:4840 , piece_id=SYZKKMI2NKG3I7C4YYRC25QVLO5CMOASWYOE75DAY45YPDSY7HRQ, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.530430180 (720h40m59.992522663s) [noise=true , address=145.79.227.20:4439 , piece_id=JHD5XPCBGEMDP67IFSKMZJQI7FREAFFH7N3F42MU7JYHY32T6R4Q, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.588451766 (720h41m0.050544249s) [noise=true , address=145.79.227.14:4839 , piece_id=UR62ATUIMJJQ7UZC5T7IWYA5Z35STVWECHG7HY6OJGYHYHJ467JQ, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.588973710 (720h41m0.051066193s) [noise=true , address=145.79.227.19:4077 , piece_id=QQCCQ2OHYBHEFER257M7ZTAE7FTOQKZM6EFKZOAJZMTUI776XRVQ, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.603144364 (720h41m0.065236847s) [noise=true , address=145.79.227.26:4654 , piece_id=XDFYPG5ILF2UFYK76M455LWP5TWYIMAYWINUXYUHOV6ETL2DGG4Q, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.615118234 (720h41m0.077210717s) [noise=true , address=145.79.227.11:4730 , piece_id=GJUKYXUPMETZCTIXQPQDPJJL3YSIPEOIO23KMBA33IQUZTV4GGEA, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.677591017 (720h41m0.1396835s) [noise=true , address=77.90.31.12:4592 , piece_id=5TVQIJTVNQHQZAS5ZD2NUH5TZINEXPYP3QYO2BNMITKN7RBIOOVA, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.748693641 (720h41m0.210786124s) [noise=true , address=45.91.116.6:4975 , piece_id=SKM7JOZ3W5RLS25ZOA6222CG7DZ3M6I2W2N2353YFHYNPTGM6L3Q, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.800596134 (720h41m0.262688617s) [noise=true , address=77.90.31.11:4553 , piece_id=WBW4C6GESNPKZLLPRAFHDG2SOUAJSYEYEYRQV4TTBXI4NFS6TNQQ, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.809229134 (720h41m0.271321617s) [noise=true , address=77.90.31.16:4735 , piece_id=5ZOPVHHAEOAIRVH2XJ3PHZUFP4YXSAU5NB5FT5RJLJNVY3D6KUDQ, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.809288828 (720h41m0.271381311s) [seg_pos={0 0}, upload=633955]: Segment reached optimal threshold of 13 pieces. Oct 23 14:12:14.809633875 (720h41m0.271726358s) [noise=true , address=77.90.31.13:4626 , piece_id=3UWMJRRQU5G6LQGTG2RABD3UE46SMUBPQIQUDYIJDCJ224J5AKSQ, seg_pos={0 0}, upload=633955]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Oct 23 14:12:14.809661617 (720h41m0.27175409s) [noise=true , address=77.90.31.13:4626 , piece_id=3UWMJRRQU5G6LQGTG2RABD3UE46SMUBPQIQUDYIJDCJ224J5AKSQ, seg_pos={0 0}, upload=633955]: Long tail cancellation due to optimal threshold Oct 23 14:12:14.809823636 (720h41m0.271916109s) [noise=true , address=145.79.227.23:4545 , piece_id=TQL2O7TQ6AX3ADEVMNSMPRLTB4LRY7IHNMDGT5RTKGIBHPYBGV6Q, seg_pos={0 0}, upload=633955]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra): protocol: write tcp 188.166.225.170:36684->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:36684->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36684->145.79.227.23:4545: use of closed network connection Oct 23 14:12:14.809834356 (720h41m0.271926829s) [noise=true , address=145.79.227.23:4545 , piece_id=TQL2O7TQ6AX3ADEVMNSMPRLTB4LRY7IHNMDGT5RTKGIBHPYBGV6Q, seg_pos={0 0}, upload=633955]: Long tail cancellation due to optimal threshold Oct 23 14:12:14.809966377 (720h41m0.27205886s) [noise=true , address=77.90.31.10:4517 , piece_id=XJFYCOQDGEJN2YW45GSRURMYRBIVJDXVN4JDBZHDZZBBDK5W2AAQ, seg_pos={0 0}, upload=633955]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Oct 23 14:12:14.809979944 (720h41m0.272072427s) [noise=true , address=77.90.31.10:4517 , piece_id=XJFYCOQDGEJN2YW45GSRURMYRBIVJDXVN4JDBZHDZZBBDK5W2AAQ, seg_pos={0 0}, upload=633955]: Long tail cancellation due to optimal threshold Oct 23 14:12:14.810045387 (720h41m0.272137871s) [seg_pos={0 0}, upload=633955]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:12:14.810271368 (720h41m0.272363861s) [upload=633955]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:12:14.810281527 (720h41m0.27237401s) [upload=633955]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:12:15.146250265 (720h41m0.608342758s) [upload=633955]: Done uploading segments... Oct 23 14:12:15.146275132 (720h41m0.608367615s) [upload=633955]: Upload finished. err: <nil> Duration: 6.119385199s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.040946914s FAIL (2025-10-23 14:17:14 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 2.040971822s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 10.640570211s FAIL (2025-10-23 14:16:57 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 10.640598857s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.878989745s FAIL (2025-10-23 14:16:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:12:11.907236120 (720h40m57.369328593s) [upload=633955]: Starting upload Oct 23 14:12:11.907307306 (720h40m57.369399789s) [upload=633955]: Done starting upload Oct 23 14:12:11.907391667 (720h40m57.36948415s) [upload=633955]: Uploading segments... Oct 23 14:12:11.907493661 (720h40m57.369586154s) [upload=633955]: Got next segment. Inline: false Oct 23 14:12:11.907561601 (720h40m57.369654084s) [upload=633955]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:12:11.907568914 (720h40m57.369661417s) [upload=633955]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:12:12.459592256 (720h40m57.921684739s) [seg_pos={0 0}, upload=633955]: Begin upload segment... Oct 23 14:12:12.459664634 (720h40m57.921757117s) [seg_pos={0 0}, upload=633955]: Done begin upload segment. Oct 23 14:12:12.459686255 (720h40m57.921778728s) [noise=true , address=145.79.227.3:4150 , piece_id=LDZ4NVA7GSORO53QARUCDMJIOR5WGVIZJEQ5KAZRTDF4Z5764DTA, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.459730529 (720h40m57.921823012s) [noise=true , address=145.79.227.16:4909 , piece_id=IYIYJMIHNE62CBOEQF4N6PGGX3OICHC2OGPBQ7IDULBD2SFTHG6Q, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.459866769 (720h40m57.921959252s) [noise=true , address=145.79.227.19:4077 , piece_id=QQCCQ2OHYBHEFER257M7ZTAE7FTOQKZM6EFKZOAJZMTUI776XRVQ, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.460667133 (720h40m57.922759616s) [noise=true , address=77.90.31.12:4592 , piece_id=5TVQIJTVNQHQZAS5ZD2NUH5TZINEXPYP3QYO2BNMITKN7RBIOOVA, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.461059931 (720h40m57.923152414s) [noise=true , address=77.90.31.13:4626 , piece_id=3UWMJRRQU5G6LQGTG2RABD3UE46SMUBPQIQUDYIJDCJ224J5AKSQ, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.461441607 (720h40m57.923534091s) [noise=true , address=77.90.31.11:4553 , piece_id=WBW4C6GESNPKZLLPRAFHDG2SOUAJSYEYEYRQV4TTBXI4NFS6TNQQ, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.462275826 (720h40m57.924368309s) [noise=true , address=145.79.227.14:4840 , piece_id=SYZKKMI2NKG3I7C4YYRC25QVLO5CMOASWYOE75DAY45YPDSY7HRQ, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.462659697 (720h40m57.92475218s) [noise=true , address=145.79.227.26:4654 , piece_id=XDFYPG5ILF2UFYK76M455LWP5TWYIMAYWINUXYUHOV6ETL2DGG4Q, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.463142947 (720h40m57.92523542s) [noise=true , address=145.79.227.23:4545 , piece_id=TQL2O7TQ6AX3ADEVMNSMPRLTB4LRY7IHNMDGT5RTKGIBHPYBGV6Q, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.463514605 (720h40m57.925607078s) [noise=true , address=77.90.31.16:4735 , piece_id=5ZOPVHHAEOAIRVH2XJ3PHZUFP4YXSAU5NB5FT5RJLJNVY3D6KUDQ, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.464034425 (720h40m57.926126898s) [noise=true , address=145.79.227.20:4439 , piece_id=JHD5XPCBGEMDP67IFSKMZJQI7FREAFFH7N3F42MU7JYHY32T6R4Q, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.464441741 (720h40m57.926534214s) [noise=true , address=145.79.227.7:4294 , piece_id=32JC2H25VALZ3SQFD4XUK2HZSR437WUNIRE7TTBAU2NEKVCE5L4Q, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.464628176 (720h40m57.926720659s) [noise=true , address=77.90.31.10:4517 , piece_id=XJFYCOQDGEJN2YW45GSRURMYRBIVJDXVN4JDBZHDZZBBDK5W2AAQ, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.465067192 (720h40m57.927159675s) [noise=true , address=145.79.227.14:4839 , piece_id=UR62ATUIMJJQ7UZC5T7IWYA5Z35STVWECHG7HY6OJGYHYHJ467JQ, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.468800320 (720h40m57.930892803s) [noise=true , address=145.79.227.11:4730 , piece_id=GJUKYXUPMETZCTIXQPQDPJJL3YSIPEOIO23KMBA33IQUZTV4GGEA, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.470230053 (720h40m57.932322536s) [noise=true , address=45.91.116.6:4975 , piece_id=SKM7JOZ3W5RLS25ZOA6222CG7DZ3M6I2W2N2353YFHYNPTGM6L3Q, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:14.111778086 (720h40m59.573870569s) [upload=633955]: Next returned nil segment Oct 23 14:12:14.111799357 (720h40m59.57389184s) [upload=633955]: Waiting for error group managing segments... Oct 23 14:12:14.498715648 (720h40m59.960808132s) [noise=true , address=145.79.227.3:4150 , piece_id=LDZ4NVA7GSORO53QARUCDMJIOR5WGVIZJEQ5KAZRTDF4Z5764DTA, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.510890272 (720h40m59.972982755s) [noise=true , address=145.79.227.16:4909 , piece_id=IYIYJMIHNE62CBOEQF4N6PGGX3OICHC2OGPBQ7IDULBD2SFTHG6Q, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.511458444 (720h40m59.973550927s) [noise=true , address=145.79.227.7:4294 , piece_id=32JC2H25VALZ3SQFD4XUK2HZSR437WUNIRE7TTBAU2NEKVCE5L4Q, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.526524743 (720h40m59.988617226s) [noise=true , address=145.79.227.14:4840 , piece_id=SYZKKMI2NKG3I7C4YYRC25QVLO5CMOASWYOE75DAY45YPDSY7HRQ, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.530430180 (720h40m59.992522663s) [noise=true , address=145.79.227.20:4439 , piece_id=JHD5XPCBGEMDP67IFSKMZJQI7FREAFFH7N3F42MU7JYHY32T6R4Q, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.588451766 (720h41m0.050544249s) [noise=true , address=145.79.227.14:4839 , piece_id=UR62ATUIMJJQ7UZC5T7IWYA5Z35STVWECHG7HY6OJGYHYHJ467JQ, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.588973710 (720h41m0.051066193s) [noise=true , address=145.79.227.19:4077 , piece_id=QQCCQ2OHYBHEFER257M7ZTAE7FTOQKZM6EFKZOAJZMTUI776XRVQ, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.603144364 (720h41m0.065236847s) [noise=true , address=145.79.227.26:4654 , piece_id=XDFYPG5ILF2UFYK76M455LWP5TWYIMAYWINUXYUHOV6ETL2DGG4Q, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.615118234 (720h41m0.077210717s) [noise=true , address=145.79.227.11:4730 , piece_id=GJUKYXUPMETZCTIXQPQDPJJL3YSIPEOIO23KMBA33IQUZTV4GGEA, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.677591017 (720h41m0.1396835s) [noise=true , address=77.90.31.12:4592 , piece_id=5TVQIJTVNQHQZAS5ZD2NUH5TZINEXPYP3QYO2BNMITKN7RBIOOVA, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.748693641 (720h41m0.210786124s) [noise=true , address=45.91.116.6:4975 , piece_id=SKM7JOZ3W5RLS25ZOA6222CG7DZ3M6I2W2N2353YFHYNPTGM6L3Q, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.800596134 (720h41m0.262688617s) [noise=true , address=77.90.31.11:4553 , piece_id=WBW4C6GESNPKZLLPRAFHDG2SOUAJSYEYEYRQV4TTBXI4NFS6TNQQ, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.809229134 (720h41m0.271321617s) [noise=true , address=77.90.31.16:4735 , piece_id=5ZOPVHHAEOAIRVH2XJ3PHZUFP4YXSAU5NB5FT5RJLJNVY3D6KUDQ, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.809288828 (720h41m0.271381311s) [seg_pos={0 0}, upload=633955]: Segment reached optimal threshold of 13 pieces. Oct 23 14:12:14.809633875 (720h41m0.271726358s) [noise=true , address=77.90.31.13:4626 , piece_id=3UWMJRRQU5G6LQGTG2RABD3UE46SMUBPQIQUDYIJDCJ224J5AKSQ, seg_pos={0 0}, upload=633955]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Oct 23 14:12:14.809661617 (720h41m0.27175409s) [noise=true , address=77.90.31.13:4626 , piece_id=3UWMJRRQU5G6LQGTG2RABD3UE46SMUBPQIQUDYIJDCJ224J5AKSQ, seg_pos={0 0}, upload=633955]: Long tail cancellation due to optimal threshold Oct 23 14:12:14.809823636 (720h41m0.271916109s) [noise=true , address=145.79.227.23:4545 , piece_id=TQL2O7TQ6AX3ADEVMNSMPRLTB4LRY7IHNMDGT5RTKGIBHPYBGV6Q, seg_pos={0 0}, upload=633955]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra): protocol: write tcp 188.166.225.170:36684->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:36684->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36684->145.79.227.23:4545: use of closed network connection Oct 23 14:12:14.809834356 (720h41m0.271926829s) [noise=true , address=145.79.227.23:4545 , piece_id=TQL2O7TQ6AX3ADEVMNSMPRLTB4LRY7IHNMDGT5RTKGIBHPYBGV6Q, seg_pos={0 0}, upload=633955]: Long tail cancellation due to optimal threshold Oct 23 14:12:14.809966377 (720h41m0.27205886s) [noise=true , address=77.90.31.10:4517 , piece_id=XJFYCOQDGEJN2YW45GSRURMYRBIVJDXVN4JDBZHDZZBBDK5W2AAQ, seg_pos={0 0}, upload=633955]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Oct 23 14:12:14.809979944 (720h41m0.272072427s) [noise=true , address=77.90.31.10:4517 , piece_id=XJFYCOQDGEJN2YW45GSRURMYRBIVJDXVN4JDBZHDZZBBDK5W2AAQ, seg_pos={0 0}, upload=633955]: Long tail cancellation due to optimal threshold Oct 23 14:12:14.810045387 (720h41m0.272137871s) [seg_pos={0 0}, upload=633955]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:12:14.810271368 (720h41m0.272363861s) [upload=633955]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:12:14.810281527 (720h41m0.27237401s) [upload=633955]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:12:15.146250265 (720h41m0.608342758s) [upload=633955]: Done uploading segments... Oct 23 14:12:15.146275132 (720h41m0.608367615s) [upload=633955]: Upload finished. err: <nil> Duration: 5.879021475s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.949102725s FAIL (2025-10-23 14:16:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:12:11.907236120 (720h40m57.369328593s) [upload=633955]: Starting upload Oct 23 14:12:11.907307306 (720h40m57.369399789s) [upload=633955]: Done starting upload Oct 23 14:12:11.907391667 (720h40m57.36948415s) [upload=633955]: Uploading segments... Oct 23 14:12:11.907493661 (720h40m57.369586154s) [upload=633955]: Got next segment. Inline: false Oct 23 14:12:11.907561601 (720h40m57.369654084s) [upload=633955]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:12:11.907568914 (720h40m57.369661417s) [upload=633955]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:12:12.459592256 (720h40m57.921684739s) [seg_pos={0 0}, upload=633955]: Begin upload segment... Oct 23 14:12:12.459664634 (720h40m57.921757117s) [seg_pos={0 0}, upload=633955]: Done begin upload segment. Oct 23 14:12:12.459686255 (720h40m57.921778728s) [noise=true , address=145.79.227.3:4150 , piece_id=LDZ4NVA7GSORO53QARUCDMJIOR5WGVIZJEQ5KAZRTDF4Z5764DTA, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.459730529 (720h40m57.921823012s) [noise=true , address=145.79.227.16:4909 , piece_id=IYIYJMIHNE62CBOEQF4N6PGGX3OICHC2OGPBQ7IDULBD2SFTHG6Q, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.459866769 (720h40m57.921959252s) [noise=true , address=145.79.227.19:4077 , piece_id=QQCCQ2OHYBHEFER257M7ZTAE7FTOQKZM6EFKZOAJZMTUI776XRVQ, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.460667133 (720h40m57.922759616s) [noise=true , address=77.90.31.12:4592 , piece_id=5TVQIJTVNQHQZAS5ZD2NUH5TZINEXPYP3QYO2BNMITKN7RBIOOVA, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.461059931 (720h40m57.923152414s) [noise=true , address=77.90.31.13:4626 , piece_id=3UWMJRRQU5G6LQGTG2RABD3UE46SMUBPQIQUDYIJDCJ224J5AKSQ, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.461441607 (720h40m57.923534091s) [noise=true , address=77.90.31.11:4553 , piece_id=WBW4C6GESNPKZLLPRAFHDG2SOUAJSYEYEYRQV4TTBXI4NFS6TNQQ, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.462275826 (720h40m57.924368309s) [noise=true , address=145.79.227.14:4840 , piece_id=SYZKKMI2NKG3I7C4YYRC25QVLO5CMOASWYOE75DAY45YPDSY7HRQ, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.462659697 (720h40m57.92475218s) [noise=true , address=145.79.227.26:4654 , piece_id=XDFYPG5ILF2UFYK76M455LWP5TWYIMAYWINUXYUHOV6ETL2DGG4Q, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.463142947 (720h40m57.92523542s) [noise=true , address=145.79.227.23:4545 , piece_id=TQL2O7TQ6AX3ADEVMNSMPRLTB4LRY7IHNMDGT5RTKGIBHPYBGV6Q, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.463514605 (720h40m57.925607078s) [noise=true , address=77.90.31.16:4735 , piece_id=5ZOPVHHAEOAIRVH2XJ3PHZUFP4YXSAU5NB5FT5RJLJNVY3D6KUDQ, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.464034425 (720h40m57.926126898s) [noise=true , address=145.79.227.20:4439 , piece_id=JHD5XPCBGEMDP67IFSKMZJQI7FREAFFH7N3F42MU7JYHY32T6R4Q, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.464441741 (720h40m57.926534214s) [noise=true , address=145.79.227.7:4294 , piece_id=32JC2H25VALZ3SQFD4XUK2HZSR437WUNIRE7TTBAU2NEKVCE5L4Q, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.464628176 (720h40m57.926720659s) [noise=true , address=77.90.31.10:4517 , piece_id=XJFYCOQDGEJN2YW45GSRURMYRBIVJDXVN4JDBZHDZZBBDK5W2AAQ, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.465067192 (720h40m57.927159675s) [noise=true , address=145.79.227.14:4839 , piece_id=UR62ATUIMJJQ7UZC5T7IWYA5Z35STVWECHG7HY6OJGYHYHJ467JQ, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.468800320 (720h40m57.930892803s) [noise=true , address=145.79.227.11:4730 , piece_id=GJUKYXUPMETZCTIXQPQDPJJL3YSIPEOIO23KMBA33IQUZTV4GGEA, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.470230053 (720h40m57.932322536s) [noise=true , address=45.91.116.6:4975 , piece_id=SKM7JOZ3W5RLS25ZOA6222CG7DZ3M6I2W2N2353YFHYNPTGM6L3Q, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:14.111778086 (720h40m59.573870569s) [upload=633955]: Next returned nil segment Oct 23 14:12:14.111799357 (720h40m59.57389184s) [upload=633955]: Waiting for error group managing segments... Oct 23 14:12:14.498715648 (720h40m59.960808132s) [noise=true , address=145.79.227.3:4150 , piece_id=LDZ4NVA7GSORO53QARUCDMJIOR5WGVIZJEQ5KAZRTDF4Z5764DTA, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.510890272 (720h40m59.972982755s) [noise=true , address=145.79.227.16:4909 , piece_id=IYIYJMIHNE62CBOEQF4N6PGGX3OICHC2OGPBQ7IDULBD2SFTHG6Q, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.511458444 (720h40m59.973550927s) [noise=true , address=145.79.227.7:4294 , piece_id=32JC2H25VALZ3SQFD4XUK2HZSR437WUNIRE7TTBAU2NEKVCE5L4Q, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.526524743 (720h40m59.988617226s) [noise=true , address=145.79.227.14:4840 , piece_id=SYZKKMI2NKG3I7C4YYRC25QVLO5CMOASWYOE75DAY45YPDSY7HRQ, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.530430180 (720h40m59.992522663s) [noise=true , address=145.79.227.20:4439 , piece_id=JHD5XPCBGEMDP67IFSKMZJQI7FREAFFH7N3F42MU7JYHY32T6R4Q, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.588451766 (720h41m0.050544249s) [noise=true , address=145.79.227.14:4839 , piece_id=UR62ATUIMJJQ7UZC5T7IWYA5Z35STVWECHG7HY6OJGYHYHJ467JQ, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.588973710 (720h41m0.051066193s) [noise=true , address=145.79.227.19:4077 , piece_id=QQCCQ2OHYBHEFER257M7ZTAE7FTOQKZM6EFKZOAJZMTUI776XRVQ, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.603144364 (720h41m0.065236847s) [noise=true , address=145.79.227.26:4654 , piece_id=XDFYPG5ILF2UFYK76M455LWP5TWYIMAYWINUXYUHOV6ETL2DGG4Q, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.615118234 (720h41m0.077210717s) [noise=true , address=145.79.227.11:4730 , piece_id=GJUKYXUPMETZCTIXQPQDPJJL3YSIPEOIO23KMBA33IQUZTV4GGEA, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.677591017 (720h41m0.1396835s) [noise=true , address=77.90.31.12:4592 , piece_id=5TVQIJTVNQHQZAS5ZD2NUH5TZINEXPYP3QYO2BNMITKN7RBIOOVA, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.748693641 (720h41m0.210786124s) [noise=true , address=45.91.116.6:4975 , piece_id=SKM7JOZ3W5RLS25ZOA6222CG7DZ3M6I2W2N2353YFHYNPTGM6L3Q, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.800596134 (720h41m0.262688617s) [noise=true , address=77.90.31.11:4553 , piece_id=WBW4C6GESNPKZLLPRAFHDG2SOUAJSYEYEYRQV4TTBXI4NFS6TNQQ, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.809229134 (720h41m0.271321617s) [noise=true , address=77.90.31.16:4735 , piece_id=5ZOPVHHAEOAIRVH2XJ3PHZUFP4YXSAU5NB5FT5RJLJNVY3D6KUDQ, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.809288828 (720h41m0.271381311s) [seg_pos={0 0}, upload=633955]: Segment reached optimal threshold of 13 pieces. Oct 23 14:12:14.809633875 (720h41m0.271726358s) [noise=true , address=77.90.31.13:4626 , piece_id=3UWMJRRQU5G6LQGTG2RABD3UE46SMUBPQIQUDYIJDCJ224J5AKSQ, seg_pos={0 0}, upload=633955]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Oct 23 14:12:14.809661617 (720h41m0.27175409s) [noise=true , address=77.90.31.13:4626 , piece_id=3UWMJRRQU5G6LQGTG2RABD3UE46SMUBPQIQUDYIJDCJ224J5AKSQ, seg_pos={0 0}, upload=633955]: Long tail cancellation due to optimal threshold Oct 23 14:12:14.809823636 (720h41m0.271916109s) [noise=true , address=145.79.227.23:4545 , piece_id=TQL2O7TQ6AX3ADEVMNSMPRLTB4LRY7IHNMDGT5RTKGIBHPYBGV6Q, seg_pos={0 0}, upload=633955]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra): protocol: write tcp 188.166.225.170:36684->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:36684->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36684->145.79.227.23:4545: use of closed network connection Oct 23 14:12:14.809834356 (720h41m0.271926829s) [noise=true , address=145.79.227.23:4545 , piece_id=TQL2O7TQ6AX3ADEVMNSMPRLTB4LRY7IHNMDGT5RTKGIBHPYBGV6Q, seg_pos={0 0}, upload=633955]: Long tail cancellation due to optimal threshold Oct 23 14:12:14.809966377 (720h41m0.27205886s) [noise=true , address=77.90.31.10:4517 , piece_id=XJFYCOQDGEJN2YW45GSRURMYRBIVJDXVN4JDBZHDZZBBDK5W2AAQ, seg_pos={0 0}, upload=633955]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Oct 23 14:12:14.809979944 (720h41m0.272072427s) [noise=true , address=77.90.31.10:4517 , piece_id=XJFYCOQDGEJN2YW45GSRURMYRBIVJDXVN4JDBZHDZZBBDK5W2AAQ, seg_pos={0 0}, upload=633955]: Long tail cancellation due to optimal threshold Oct 23 14:12:14.810045387 (720h41m0.272137871s) [seg_pos={0 0}, upload=633955]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:12:14.810271368 (720h41m0.272363861s) [upload=633955]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:12:14.810281527 (720h41m0.27237401s) [upload=633955]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:12:15.146250265 (720h41m0.608342758s) [upload=633955]: Done uploading segments... Oct 23 14:12:15.146275132 (720h41m0.608367615s) [upload=633955]: Upload finished. err: <nil> Duration: 5.949134217s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.491045635s FAIL (2025-10-23 14:16:22 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.491565405s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.037466404s FAIL (2025-10-23 14:16:09 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 2.037499086s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.884068897s FAIL (2025-10-23 14:15:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:12:11.907236120 (720h40m57.369328593s) [upload=633955]: Starting upload Oct 23 14:12:11.907307306 (720h40m57.369399789s) [upload=633955]: Done starting upload Oct 23 14:12:11.907391667 (720h40m57.36948415s) [upload=633955]: Uploading segments... Oct 23 14:12:11.907493661 (720h40m57.369586154s) [upload=633955]: Got next segment. Inline: false Oct 23 14:12:11.907561601 (720h40m57.369654084s) [upload=633955]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:12:11.907568914 (720h40m57.369661417s) [upload=633955]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:12:12.459592256 (720h40m57.921684739s) [seg_pos={0 0}, upload=633955]: Begin upload segment... Oct 23 14:12:12.459664634 (720h40m57.921757117s) [seg_pos={0 0}, upload=633955]: Done begin upload segment. Oct 23 14:12:12.459686255 (720h40m57.921778728s) [noise=true , address=145.79.227.3:4150 , piece_id=LDZ4NVA7GSORO53QARUCDMJIOR5WGVIZJEQ5KAZRTDF4Z5764DTA, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.459730529 (720h40m57.921823012s) [noise=true , address=145.79.227.16:4909 , piece_id=IYIYJMIHNE62CBOEQF4N6PGGX3OICHC2OGPBQ7IDULBD2SFTHG6Q, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.459866769 (720h40m57.921959252s) [noise=true , address=145.79.227.19:4077 , piece_id=QQCCQ2OHYBHEFER257M7ZTAE7FTOQKZM6EFKZOAJZMTUI776XRVQ, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.460667133 (720h40m57.922759616s) [noise=true , address=77.90.31.12:4592 , piece_id=5TVQIJTVNQHQZAS5ZD2NUH5TZINEXPYP3QYO2BNMITKN7RBIOOVA, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.461059931 (720h40m57.923152414s) [noise=true , address=77.90.31.13:4626 , piece_id=3UWMJRRQU5G6LQGTG2RABD3UE46SMUBPQIQUDYIJDCJ224J5AKSQ, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.461441607 (720h40m57.923534091s) [noise=true , address=77.90.31.11:4553 , piece_id=WBW4C6GESNPKZLLPRAFHDG2SOUAJSYEYEYRQV4TTBXI4NFS6TNQQ, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.462275826 (720h40m57.924368309s) [noise=true , address=145.79.227.14:4840 , piece_id=SYZKKMI2NKG3I7C4YYRC25QVLO5CMOASWYOE75DAY45YPDSY7HRQ, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.462659697 (720h40m57.92475218s) [noise=true , address=145.79.227.26:4654 , piece_id=XDFYPG5ILF2UFYK76M455LWP5TWYIMAYWINUXYUHOV6ETL2DGG4Q, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.463142947 (720h40m57.92523542s) [noise=true , address=145.79.227.23:4545 , piece_id=TQL2O7TQ6AX3ADEVMNSMPRLTB4LRY7IHNMDGT5RTKGIBHPYBGV6Q, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.463514605 (720h40m57.925607078s) [noise=true , address=77.90.31.16:4735 , piece_id=5ZOPVHHAEOAIRVH2XJ3PHZUFP4YXSAU5NB5FT5RJLJNVY3D6KUDQ, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.464034425 (720h40m57.926126898s) [noise=true , address=145.79.227.20:4439 , piece_id=JHD5XPCBGEMDP67IFSKMZJQI7FREAFFH7N3F42MU7JYHY32T6R4Q, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.464441741 (720h40m57.926534214s) [noise=true , address=145.79.227.7:4294 , piece_id=32JC2H25VALZ3SQFD4XUK2HZSR437WUNIRE7TTBAU2NEKVCE5L4Q, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.464628176 (720h40m57.926720659s) [noise=true , address=77.90.31.10:4517 , piece_id=XJFYCOQDGEJN2YW45GSRURMYRBIVJDXVN4JDBZHDZZBBDK5W2AAQ, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.465067192 (720h40m57.927159675s) [noise=true , address=145.79.227.14:4839 , piece_id=UR62ATUIMJJQ7UZC5T7IWYA5Z35STVWECHG7HY6OJGYHYHJ467JQ, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.468800320 (720h40m57.930892803s) [noise=true , address=145.79.227.11:4730 , piece_id=GJUKYXUPMETZCTIXQPQDPJJL3YSIPEOIO23KMBA33IQUZTV4GGEA, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.470230053 (720h40m57.932322536s) [noise=true , address=45.91.116.6:4975 , piece_id=SKM7JOZ3W5RLS25ZOA6222CG7DZ3M6I2W2N2353YFHYNPTGM6L3Q, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:14.111778086 (720h40m59.573870569s) [upload=633955]: Next returned nil segment Oct 23 14:12:14.111799357 (720h40m59.57389184s) [upload=633955]: Waiting for error group managing segments... Oct 23 14:12:14.498715648 (720h40m59.960808132s) [noise=true , address=145.79.227.3:4150 , piece_id=LDZ4NVA7GSORO53QARUCDMJIOR5WGVIZJEQ5KAZRTDF4Z5764DTA, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.510890272 (720h40m59.972982755s) [noise=true , address=145.79.227.16:4909 , piece_id=IYIYJMIHNE62CBOEQF4N6PGGX3OICHC2OGPBQ7IDULBD2SFTHG6Q, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.511458444 (720h40m59.973550927s) [noise=true , address=145.79.227.7:4294 , piece_id=32JC2H25VALZ3SQFD4XUK2HZSR437WUNIRE7TTBAU2NEKVCE5L4Q, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.526524743 (720h40m59.988617226s) [noise=true , address=145.79.227.14:4840 , piece_id=SYZKKMI2NKG3I7C4YYRC25QVLO5CMOASWYOE75DAY45YPDSY7HRQ, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.530430180 (720h40m59.992522663s) [noise=true , address=145.79.227.20:4439 , piece_id=JHD5XPCBGEMDP67IFSKMZJQI7FREAFFH7N3F42MU7JYHY32T6R4Q, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.588451766 (720h41m0.050544249s) [noise=true , address=145.79.227.14:4839 , piece_id=UR62ATUIMJJQ7UZC5T7IWYA5Z35STVWECHG7HY6OJGYHYHJ467JQ, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.588973710 (720h41m0.051066193s) [noise=true , address=145.79.227.19:4077 , piece_id=QQCCQ2OHYBHEFER257M7ZTAE7FTOQKZM6EFKZOAJZMTUI776XRVQ, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.603144364 (720h41m0.065236847s) [noise=true , address=145.79.227.26:4654 , piece_id=XDFYPG5ILF2UFYK76M455LWP5TWYIMAYWINUXYUHOV6ETL2DGG4Q, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.615118234 (720h41m0.077210717s) [noise=true , address=145.79.227.11:4730 , piece_id=GJUKYXUPMETZCTIXQPQDPJJL3YSIPEOIO23KMBA33IQUZTV4GGEA, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.677591017 (720h41m0.1396835s) [noise=true , address=77.90.31.12:4592 , piece_id=5TVQIJTVNQHQZAS5ZD2NUH5TZINEXPYP3QYO2BNMITKN7RBIOOVA, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.748693641 (720h41m0.210786124s) [noise=true , address=45.91.116.6:4975 , piece_id=SKM7JOZ3W5RLS25ZOA6222CG7DZ3M6I2W2N2353YFHYNPTGM6L3Q, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.800596134 (720h41m0.262688617s) [noise=true , address=77.90.31.11:4553 , piece_id=WBW4C6GESNPKZLLPRAFHDG2SOUAJSYEYEYRQV4TTBXI4NFS6TNQQ, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.809229134 (720h41m0.271321617s) [noise=true , address=77.90.31.16:4735 , piece_id=5ZOPVHHAEOAIRVH2XJ3PHZUFP4YXSAU5NB5FT5RJLJNVY3D6KUDQ, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.809288828 (720h41m0.271381311s) [seg_pos={0 0}, upload=633955]: Segment reached optimal threshold of 13 pieces. Oct 23 14:12:14.809633875 (720h41m0.271726358s) [noise=true , address=77.90.31.13:4626 , piece_id=3UWMJRRQU5G6LQGTG2RABD3UE46SMUBPQIQUDYIJDCJ224J5AKSQ, seg_pos={0 0}, upload=633955]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Oct 23 14:12:14.809661617 (720h41m0.27175409s) [noise=true , address=77.90.31.13:4626 , piece_id=3UWMJRRQU5G6LQGTG2RABD3UE46SMUBPQIQUDYIJDCJ224J5AKSQ, seg_pos={0 0}, upload=633955]: Long tail cancellation due to optimal threshold Oct 23 14:12:14.809823636 (720h41m0.271916109s) [noise=true , address=145.79.227.23:4545 , piece_id=TQL2O7TQ6AX3ADEVMNSMPRLTB4LRY7IHNMDGT5RTKGIBHPYBGV6Q, seg_pos={0 0}, upload=633955]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra): protocol: write tcp 188.166.225.170:36684->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:36684->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36684->145.79.227.23:4545: use of closed network connection Oct 23 14:12:14.809834356 (720h41m0.271926829s) [noise=true , address=145.79.227.23:4545 , piece_id=TQL2O7TQ6AX3ADEVMNSMPRLTB4LRY7IHNMDGT5RTKGIBHPYBGV6Q, seg_pos={0 0}, upload=633955]: Long tail cancellation due to optimal threshold Oct 23 14:12:14.809966377 (720h41m0.27205886s) [noise=true , address=77.90.31.10:4517 , piece_id=XJFYCOQDGEJN2YW45GSRURMYRBIVJDXVN4JDBZHDZZBBDK5W2AAQ, seg_pos={0 0}, upload=633955]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Oct 23 14:12:14.809979944 (720h41m0.272072427s) [noise=true , address=77.90.31.10:4517 , piece_id=XJFYCOQDGEJN2YW45GSRURMYRBIVJDXVN4JDBZHDZZBBDK5W2AAQ, seg_pos={0 0}, upload=633955]: Long tail cancellation due to optimal threshold Oct 23 14:12:14.810045387 (720h41m0.272137871s) [seg_pos={0 0}, upload=633955]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:12:14.810271368 (720h41m0.272363861s) [upload=633955]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:12:14.810281527 (720h41m0.27237401s) [upload=633955]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:12:15.146250265 (720h41m0.608342758s) [upload=633955]: Done uploading segments... Oct 23 14:12:15.146275132 (720h41m0.608367615s) [upload=633955]: Upload finished. err: <nil> Duration: 5.884098313s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.06899343s FAIL (2025-10-23 14:15:36 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 2.069046863s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.960276896s FAIL (2025-10-23 14:15:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:12:11.907236120 (720h40m57.369328593s) [upload=633955]: Starting upload Oct 23 14:12:11.907307306 (720h40m57.369399789s) [upload=633955]: Done starting upload Oct 23 14:12:11.907391667 (720h40m57.36948415s) [upload=633955]: Uploading segments... Oct 23 14:12:11.907493661 (720h40m57.369586154s) [upload=633955]: Got next segment. Inline: false Oct 23 14:12:11.907561601 (720h40m57.369654084s) [upload=633955]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:12:11.907568914 (720h40m57.369661417s) [upload=633955]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:12:12.459592256 (720h40m57.921684739s) [seg_pos={0 0}, upload=633955]: Begin upload segment... Oct 23 14:12:12.459664634 (720h40m57.921757117s) [seg_pos={0 0}, upload=633955]: Done begin upload segment. Oct 23 14:12:12.459686255 (720h40m57.921778728s) [noise=true , address=145.79.227.3:4150 , piece_id=LDZ4NVA7GSORO53QARUCDMJIOR5WGVIZJEQ5KAZRTDF4Z5764DTA, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.459730529 (720h40m57.921823012s) [noise=true , address=145.79.227.16:4909 , piece_id=IYIYJMIHNE62CBOEQF4N6PGGX3OICHC2OGPBQ7IDULBD2SFTHG6Q, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.459866769 (720h40m57.921959252s) [noise=true , address=145.79.227.19:4077 , piece_id=QQCCQ2OHYBHEFER257M7ZTAE7FTOQKZM6EFKZOAJZMTUI776XRVQ, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.460667133 (720h40m57.922759616s) [noise=true , address=77.90.31.12:4592 , piece_id=5TVQIJTVNQHQZAS5ZD2NUH5TZINEXPYP3QYO2BNMITKN7RBIOOVA, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.461059931 (720h40m57.923152414s) [noise=true , address=77.90.31.13:4626 , piece_id=3UWMJRRQU5G6LQGTG2RABD3UE46SMUBPQIQUDYIJDCJ224J5AKSQ, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.461441607 (720h40m57.923534091s) [noise=true , address=77.90.31.11:4553 , piece_id=WBW4C6GESNPKZLLPRAFHDG2SOUAJSYEYEYRQV4TTBXI4NFS6TNQQ, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.462275826 (720h40m57.924368309s) [noise=true , address=145.79.227.14:4840 , piece_id=SYZKKMI2NKG3I7C4YYRC25QVLO5CMOASWYOE75DAY45YPDSY7HRQ, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.462659697 (720h40m57.92475218s) [noise=true , address=145.79.227.26:4654 , piece_id=XDFYPG5ILF2UFYK76M455LWP5TWYIMAYWINUXYUHOV6ETL2DGG4Q, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.463142947 (720h40m57.92523542s) [noise=true , address=145.79.227.23:4545 , piece_id=TQL2O7TQ6AX3ADEVMNSMPRLTB4LRY7IHNMDGT5RTKGIBHPYBGV6Q, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.463514605 (720h40m57.925607078s) [noise=true , address=77.90.31.16:4735 , piece_id=5ZOPVHHAEOAIRVH2XJ3PHZUFP4YXSAU5NB5FT5RJLJNVY3D6KUDQ, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.464034425 (720h40m57.926126898s) [noise=true , address=145.79.227.20:4439 , piece_id=JHD5XPCBGEMDP67IFSKMZJQI7FREAFFH7N3F42MU7JYHY32T6R4Q, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.464441741 (720h40m57.926534214s) [noise=true , address=145.79.227.7:4294 , piece_id=32JC2H25VALZ3SQFD4XUK2HZSR437WUNIRE7TTBAU2NEKVCE5L4Q, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.464628176 (720h40m57.926720659s) [noise=true , address=77.90.31.10:4517 , piece_id=XJFYCOQDGEJN2YW45GSRURMYRBIVJDXVN4JDBZHDZZBBDK5W2AAQ, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.465067192 (720h40m57.927159675s) [noise=true , address=145.79.227.14:4839 , piece_id=UR62ATUIMJJQ7UZC5T7IWYA5Z35STVWECHG7HY6OJGYHYHJ467JQ, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.468800320 (720h40m57.930892803s) [noise=true , address=145.79.227.11:4730 , piece_id=GJUKYXUPMETZCTIXQPQDPJJL3YSIPEOIO23KMBA33IQUZTV4GGEA, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:12.470230053 (720h40m57.932322536s) [noise=true , address=45.91.116.6:4975 , piece_id=SKM7JOZ3W5RLS25ZOA6222CG7DZ3M6I2W2N2353YFHYNPTGM6L3Q, seg_pos={0 0}, upload=633955]: Uploading piece... Oct 23 14:12:14.111778086 (720h40m59.573870569s) [upload=633955]: Next returned nil segment Oct 23 14:12:14.111799357 (720h40m59.57389184s) [upload=633955]: Waiting for error group managing segments... Oct 23 14:12:14.498715648 (720h40m59.960808132s) [noise=true , address=145.79.227.3:4150 , piece_id=LDZ4NVA7GSORO53QARUCDMJIOR5WGVIZJEQ5KAZRTDF4Z5764DTA, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.510890272 (720h40m59.972982755s) [noise=true , address=145.79.227.16:4909 , piece_id=IYIYJMIHNE62CBOEQF4N6PGGX3OICHC2OGPBQ7IDULBD2SFTHG6Q, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.511458444 (720h40m59.973550927s) [noise=true , address=145.79.227.7:4294 , piece_id=32JC2H25VALZ3SQFD4XUK2HZSR437WUNIRE7TTBAU2NEKVCE5L4Q, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.526524743 (720h40m59.988617226s) [noise=true , address=145.79.227.14:4840 , piece_id=SYZKKMI2NKG3I7C4YYRC25QVLO5CMOASWYOE75DAY45YPDSY7HRQ, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.530430180 (720h40m59.992522663s) [noise=true , address=145.79.227.20:4439 , piece_id=JHD5XPCBGEMDP67IFSKMZJQI7FREAFFH7N3F42MU7JYHY32T6R4Q, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.588451766 (720h41m0.050544249s) [noise=true , address=145.79.227.14:4839 , piece_id=UR62ATUIMJJQ7UZC5T7IWYA5Z35STVWECHG7HY6OJGYHYHJ467JQ, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.588973710 (720h41m0.051066193s) [noise=true , address=145.79.227.19:4077 , piece_id=QQCCQ2OHYBHEFER257M7ZTAE7FTOQKZM6EFKZOAJZMTUI776XRVQ, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.603144364 (720h41m0.065236847s) [noise=true , address=145.79.227.26:4654 , piece_id=XDFYPG5ILF2UFYK76M455LWP5TWYIMAYWINUXYUHOV6ETL2DGG4Q, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.615118234 (720h41m0.077210717s) [noise=true , address=145.79.227.11:4730 , piece_id=GJUKYXUPMETZCTIXQPQDPJJL3YSIPEOIO23KMBA33IQUZTV4GGEA, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.677591017 (720h41m0.1396835s) [noise=true , address=77.90.31.12:4592 , piece_id=5TVQIJTVNQHQZAS5ZD2NUH5TZINEXPYP3QYO2BNMITKN7RBIOOVA, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.748693641 (720h41m0.210786124s) [noise=true , address=45.91.116.6:4975 , piece_id=SKM7JOZ3W5RLS25ZOA6222CG7DZ3M6I2W2N2353YFHYNPTGM6L3Q, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.800596134 (720h41m0.262688617s) [noise=true , address=77.90.31.11:4553 , piece_id=WBW4C6GESNPKZLLPRAFHDG2SOUAJSYEYEYRQV4TTBXI4NFS6TNQQ, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.809229134 (720h41m0.271321617s) [noise=true , address=77.90.31.16:4735 , piece_id=5ZOPVHHAEOAIRVH2XJ3PHZUFP4YXSAU5NB5FT5RJLJNVY3D6KUDQ, seg_pos={0 0}, upload=633955]: Done uploading piece. err: <nil> Oct 23 14:12:14.809288828 (720h41m0.271381311s) [seg_pos={0 0}, upload=633955]: Segment reached optimal threshold of 13 pieces. Oct 23 14:12:14.809633875 (720h41m0.271726358s) [noise=true , address=77.90.31.13:4626 , piece_id=3UWMJRRQU5G6LQGTG2RABD3UE46SMUBPQIQUDYIJDCJ224J5AKSQ, seg_pos={0 0}, upload=633955]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Oct 23 14:12:14.809661617 (720h41m0.27175409s) [noise=true , address=77.90.31.13:4626 , piece_id=3UWMJRRQU5G6LQGTG2RABD3UE46SMUBPQIQUDYIJDCJ224J5AKSQ, seg_pos={0 0}, upload=633955]: Long tail cancellation due to optimal threshold Oct 23 14:12:14.809823636 (720h41m0.271916109s) [noise=true , address=145.79.227.23:4545 , piece_id=TQL2O7TQ6AX3ADEVMNSMPRLTB4LRY7IHNMDGT5RTKGIBHPYBGV6Q, seg_pos={0 0}, upload=633955]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra): protocol: write tcp 188.166.225.170:36684->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:36684->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36684->145.79.227.23:4545: use of closed network connection Oct 23 14:12:14.809834356 (720h41m0.271926829s) [noise=true , address=145.79.227.23:4545 , piece_id=TQL2O7TQ6AX3ADEVMNSMPRLTB4LRY7IHNMDGT5RTKGIBHPYBGV6Q, seg_pos={0 0}, upload=633955]: Long tail cancellation due to optimal threshold Oct 23 14:12:14.809966377 (720h41m0.27205886s) [noise=true , address=77.90.31.10:4517 , piece_id=XJFYCOQDGEJN2YW45GSRURMYRBIVJDXVN4JDBZHDZZBBDK5W2AAQ, seg_pos={0 0}, upload=633955]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Oct 23 14:12:14.809979944 (720h41m0.272072427s) [noise=true , address=77.90.31.10:4517 , piece_id=XJFYCOQDGEJN2YW45GSRURMYRBIVJDXVN4JDBZHDZZBBDK5W2AAQ, seg_pos={0 0}, upload=633955]: Long tail cancellation due to optimal threshold Oct 23 14:12:14.810045387 (720h41m0.272137871s) [seg_pos={0 0}, upload=633955]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:12:14.810271368 (720h41m0.272363861s) [upload=633955]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:12:14.810281527 (720h41m0.27237401s) [upload=633955]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:12:15.146250265 (720h41m0.608342758s) [upload=633955]: Done uploading segments... Oct 23 14:12:15.146275132 (720h41m0.608367615s) [upload=633955]: Upload finished. err: <nil> Duration: 5.960309299s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.453041126s FAIL (2025-10-23 14:15:17 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.453079078s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 3.356481245s FAIL (2025-10-23 14:15:02 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 3.356524216s core uplink us1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)