Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.022116811s FAIL (2025-10-23 14:38:04 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 2.022151798s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.147859088s FAIL (2025-10-23 14:38:00 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.147887843s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X.........X...........................] duration: 6.017468803s FAIL (2025-10-23 14:37:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:18:06.304157422 (720h46m51.766249905s) [upload=634034]: Starting upload Oct 23 14:18:06.304347725 (720h46m51.766440218s) [upload=634034]: Done starting upload Oct 23 14:18:06.304632076 (720h46m51.766724559s) [upload=634034]: Uploading segments... Oct 23 14:18:06.304690778 (720h46m51.766783261s) [upload=634034]: Got next segment. Inline: false Oct 23 14:18:06.304730654 (720h46m51.766823147s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:18:06.304737897 (720h46m51.766830371s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:18:06.560110142 (720h46m52.022202615s) [seg_pos={0 0}, upload=634034]: Begin upload segment... Oct 23 14:18:06.560170466 (720h46m52.022262949s) [seg_pos={0 0}, upload=634034]: Done begin upload segment. Oct 23 14:18:06.560196877 (720h46m52.02228936s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.560384274 (720h46m52.022476767s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561087734 (720h46m52.023180217s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561246465 (720h46m52.023338948s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561388856 (720h46m52.02348134s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561682246 (720h46m52.023774729s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561840206 (720h46m52.023932699s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562036520 (720h46m52.024129003s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562369515 (720h46m52.024461998s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562402808 (720h46m52.024495301s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562804713 (720h46m52.024897196s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562996849 (720h46m52.025089332s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563336766 (720h46m52.025429249s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563574799 (720h46m52.025667282s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564175863 (720h46m52.026268346s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564848324 (720h46m52.026940798s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:08.284021691 (720h46m53.746114164s) [upload=634034]: Next returned nil segment Oct 23 14:18:08.284044505 (720h46m53.746136978s) [upload=634034]: Waiting for error group managing segments... Oct 23 14:18:08.566598218 (720h46m54.028690691s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.594453381 (720h46m54.056545864s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.598953419 (720h46m54.061045892s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.602023124 (720h46m54.064115658s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625098415 (720h46m54.087190898s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625760095 (720h46m54.087852578s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.631004962 (720h46m54.093097445s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.635035678 (720h46m54.097128161s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.645903372 (720h46m54.107995845s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.711835221 (720h46m54.173927704s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.719309664 (720h46m54.181402147s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.758298877 (720h46m54.22039136s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819795110 (720h46m54.281887593s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819860023 (720h46m54.281952516s) [seg_pos={0 0}, upload=634034]: Segment reached optimal threshold of 13 pieces. Oct 23 14:18:08.820128314 (720h46m54.282220807s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection Oct 23 14:18:08.820140978 (720h46m54.282233461s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820198287 (720h46m54.28229077s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820209729 (720h46m54.282302222s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820445267 (720h46m54.28253775s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820509670 (720h46m54.282602153s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820531111 (720h46m54.282623594s) [seg_pos={0 0}, upload=634034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:18:08.820624719 (720h46m54.282717202s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:18:08.820630982 (720h46m54.282723465s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:18:09.121056837 (720h46m54.58314933s) [upload=634034]: Done uploading segments... Oct 23 14:18:09.121104197 (720h46m54.58319668s) [upload=634034]: Upload finished. err: <nil> Duration: 6.017531482s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.010305109s FAIL (2025-10-23 14:37:42 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 2.010334734s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X.........X..........................] duration: 5.846309758s FAIL (2025-10-23 14:37:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:18:06.304157422 (720h46m51.766249905s) [upload=634034]: Starting upload Oct 23 14:18:06.304347725 (720h46m51.766440218s) [upload=634034]: Done starting upload Oct 23 14:18:06.304632076 (720h46m51.766724559s) [upload=634034]: Uploading segments... Oct 23 14:18:06.304690778 (720h46m51.766783261s) [upload=634034]: Got next segment. Inline: false Oct 23 14:18:06.304730654 (720h46m51.766823147s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:18:06.304737897 (720h46m51.766830371s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:18:06.560110142 (720h46m52.022202615s) [seg_pos={0 0}, upload=634034]: Begin upload segment... Oct 23 14:18:06.560170466 (720h46m52.022262949s) [seg_pos={0 0}, upload=634034]: Done begin upload segment. Oct 23 14:18:06.560196877 (720h46m52.02228936s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.560384274 (720h46m52.022476767s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561087734 (720h46m52.023180217s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561246465 (720h46m52.023338948s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561388856 (720h46m52.02348134s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561682246 (720h46m52.023774729s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561840206 (720h46m52.023932699s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562036520 (720h46m52.024129003s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562369515 (720h46m52.024461998s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562402808 (720h46m52.024495301s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562804713 (720h46m52.024897196s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562996849 (720h46m52.025089332s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563336766 (720h46m52.025429249s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563574799 (720h46m52.025667282s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564175863 (720h46m52.026268346s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564848324 (720h46m52.026940798s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:08.284021691 (720h46m53.746114164s) [upload=634034]: Next returned nil segment Oct 23 14:18:08.284044505 (720h46m53.746136978s) [upload=634034]: Waiting for error group managing segments... Oct 23 14:18:08.566598218 (720h46m54.028690691s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.594453381 (720h46m54.056545864s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.598953419 (720h46m54.061045892s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.602023124 (720h46m54.064115658s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625098415 (720h46m54.087190898s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625760095 (720h46m54.087852578s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.631004962 (720h46m54.093097445s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.635035678 (720h46m54.097128161s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.645903372 (720h46m54.107995845s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.711835221 (720h46m54.173927704s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.719309664 (720h46m54.181402147s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.758298877 (720h46m54.22039136s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819795110 (720h46m54.281887593s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819860023 (720h46m54.281952516s) [seg_pos={0 0}, upload=634034]: Segment reached optimal threshold of 13 pieces. Oct 23 14:18:08.820128314 (720h46m54.282220807s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection Oct 23 14:18:08.820140978 (720h46m54.282233461s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820198287 (720h46m54.28229077s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820209729 (720h46m54.282302222s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820445267 (720h46m54.28253775s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820509670 (720h46m54.282602153s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820531111 (720h46m54.282623594s) [seg_pos={0 0}, upload=634034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:18:08.820624719 (720h46m54.282717202s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:18:08.820630982 (720h46m54.282723465s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:18:09.121056837 (720h46m54.58314933s) [upload=634034]: Done uploading segments... Oct 23 14:18:09.121104197 (720h46m54.58319668s) [upload=634034]: Upload finished. err: <nil> Duration: 5.846362698s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.691017484s FAIL (2025-10-23 14:37:19 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.691054575s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 7.121316059s FAIL (2025-10-23 14:37:02 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 7.121343171s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 1.991667146s FAIL (2025-10-23 14:36:49 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 1.991709097s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X.........X.........................] duration: 5.975762624s FAIL (2025-10-23 14:36:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:18:06.304157422 (720h46m51.766249905s) [upload=634034]: Starting upload Oct 23 14:18:06.304347725 (720h46m51.766440218s) [upload=634034]: Done starting upload Oct 23 14:18:06.304632076 (720h46m51.766724559s) [upload=634034]: Uploading segments... Oct 23 14:18:06.304690778 (720h46m51.766783261s) [upload=634034]: Got next segment. Inline: false Oct 23 14:18:06.304730654 (720h46m51.766823147s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:18:06.304737897 (720h46m51.766830371s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:18:06.560110142 (720h46m52.022202615s) [seg_pos={0 0}, upload=634034]: Begin upload segment... Oct 23 14:18:06.560170466 (720h46m52.022262949s) [seg_pos={0 0}, upload=634034]: Done begin upload segment. Oct 23 14:18:06.560196877 (720h46m52.02228936s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.560384274 (720h46m52.022476767s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561087734 (720h46m52.023180217s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561246465 (720h46m52.023338948s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561388856 (720h46m52.02348134s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561682246 (720h46m52.023774729s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561840206 (720h46m52.023932699s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562036520 (720h46m52.024129003s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562369515 (720h46m52.024461998s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562402808 (720h46m52.024495301s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562804713 (720h46m52.024897196s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562996849 (720h46m52.025089332s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563336766 (720h46m52.025429249s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563574799 (720h46m52.025667282s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564175863 (720h46m52.026268346s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564848324 (720h46m52.026940798s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:08.284021691 (720h46m53.746114164s) [upload=634034]: Next returned nil segment Oct 23 14:18:08.284044505 (720h46m53.746136978s) [upload=634034]: Waiting for error group managing segments... Oct 23 14:18:08.566598218 (720h46m54.028690691s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.594453381 (720h46m54.056545864s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.598953419 (720h46m54.061045892s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.602023124 (720h46m54.064115658s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625098415 (720h46m54.087190898s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625760095 (720h46m54.087852578s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.631004962 (720h46m54.093097445s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.635035678 (720h46m54.097128161s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.645903372 (720h46m54.107995845s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.711835221 (720h46m54.173927704s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.719309664 (720h46m54.181402147s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.758298877 (720h46m54.22039136s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819795110 (720h46m54.281887593s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819860023 (720h46m54.281952516s) [seg_pos={0 0}, upload=634034]: Segment reached optimal threshold of 13 pieces. Oct 23 14:18:08.820128314 (720h46m54.282220807s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection Oct 23 14:18:08.820140978 (720h46m54.282233461s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820198287 (720h46m54.28229077s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820209729 (720h46m54.282302222s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820445267 (720h46m54.28253775s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820509670 (720h46m54.282602153s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820531111 (720h46m54.282623594s) [seg_pos={0 0}, upload=634034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:18:08.820624719 (720h46m54.282717202s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:18:08.820630982 (720h46m54.282723465s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:18:09.121056837 (720h46m54.58314933s) [upload=634034]: Done uploading segments... Oct 23 14:18:09.121104197 (720h46m54.58319668s) [upload=634034]: Upload finished. err: <nil> Duration: 5.975791327s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.769134011s FAIL (2025-10-23 14:36:26 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.769166733s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X.........X........................] duration: 5.859840871s FAIL (2025-10-23 14:35:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:18:06.304157422 (720h46m51.766249905s) [upload=634034]: Starting upload Oct 23 14:18:06.304347725 (720h46m51.766440218s) [upload=634034]: Done starting upload Oct 23 14:18:06.304632076 (720h46m51.766724559s) [upload=634034]: Uploading segments... Oct 23 14:18:06.304690778 (720h46m51.766783261s) [upload=634034]: Got next segment. Inline: false Oct 23 14:18:06.304730654 (720h46m51.766823147s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:18:06.304737897 (720h46m51.766830371s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:18:06.560110142 (720h46m52.022202615s) [seg_pos={0 0}, upload=634034]: Begin upload segment... Oct 23 14:18:06.560170466 (720h46m52.022262949s) [seg_pos={0 0}, upload=634034]: Done begin upload segment. Oct 23 14:18:06.560196877 (720h46m52.02228936s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.560384274 (720h46m52.022476767s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561087734 (720h46m52.023180217s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561246465 (720h46m52.023338948s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561388856 (720h46m52.02348134s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561682246 (720h46m52.023774729s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561840206 (720h46m52.023932699s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562036520 (720h46m52.024129003s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562369515 (720h46m52.024461998s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562402808 (720h46m52.024495301s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562804713 (720h46m52.024897196s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562996849 (720h46m52.025089332s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563336766 (720h46m52.025429249s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563574799 (720h46m52.025667282s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564175863 (720h46m52.026268346s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564848324 (720h46m52.026940798s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:08.284021691 (720h46m53.746114164s) [upload=634034]: Next returned nil segment Oct 23 14:18:08.284044505 (720h46m53.746136978s) [upload=634034]: Waiting for error group managing segments... Oct 23 14:18:08.566598218 (720h46m54.028690691s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.594453381 (720h46m54.056545864s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.598953419 (720h46m54.061045892s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.602023124 (720h46m54.064115658s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625098415 (720h46m54.087190898s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625760095 (720h46m54.087852578s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.631004962 (720h46m54.093097445s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.635035678 (720h46m54.097128161s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.645903372 (720h46m54.107995845s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.711835221 (720h46m54.173927704s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.719309664 (720h46m54.181402147s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.758298877 (720h46m54.22039136s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819795110 (720h46m54.281887593s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819860023 (720h46m54.281952516s) [seg_pos={0 0}, upload=634034]: Segment reached optimal threshold of 13 pieces. Oct 23 14:18:08.820128314 (720h46m54.282220807s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection Oct 23 14:18:08.820140978 (720h46m54.282233461s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820198287 (720h46m54.28229077s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820209729 (720h46m54.282302222s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820445267 (720h46m54.28253775s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820509670 (720h46m54.282602153s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820531111 (720h46m54.282623594s) [seg_pos={0 0}, upload=634034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:18:08.820624719 (720h46m54.282717202s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:18:08.820630982 (720h46m54.282723465s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:18:09.121056837 (720h46m54.58314933s) [upload=634034]: Done uploading segments... Oct 23 14:18:09.121104197 (720h46m54.58319668s) [upload=634034]: Upload finished. err: <nil> Duration: 5.859878072s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.066608949s FAIL (2025-10-23 14:35:50 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 2.066645918s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.62200941s FAIL (2025-10-23 14:35:42 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.622068272s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X.........X.......................] duration: 5.839399398s FAIL (2025-10-23 14:35:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:18:06.304157422 (720h46m51.766249905s) [upload=634034]: Starting upload Oct 23 14:18:06.304347725 (720h46m51.766440218s) [upload=634034]: Done starting upload Oct 23 14:18:06.304632076 (720h46m51.766724559s) [upload=634034]: Uploading segments... Oct 23 14:18:06.304690778 (720h46m51.766783261s) [upload=634034]: Got next segment. Inline: false Oct 23 14:18:06.304730654 (720h46m51.766823147s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:18:06.304737897 (720h46m51.766830371s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:18:06.560110142 (720h46m52.022202615s) [seg_pos={0 0}, upload=634034]: Begin upload segment... Oct 23 14:18:06.560170466 (720h46m52.022262949s) [seg_pos={0 0}, upload=634034]: Done begin upload segment. Oct 23 14:18:06.560196877 (720h46m52.02228936s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.560384274 (720h46m52.022476767s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561087734 (720h46m52.023180217s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561246465 (720h46m52.023338948s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561388856 (720h46m52.02348134s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561682246 (720h46m52.023774729s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561840206 (720h46m52.023932699s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562036520 (720h46m52.024129003s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562369515 (720h46m52.024461998s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562402808 (720h46m52.024495301s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562804713 (720h46m52.024897196s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562996849 (720h46m52.025089332s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563336766 (720h46m52.025429249s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563574799 (720h46m52.025667282s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564175863 (720h46m52.026268346s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564848324 (720h46m52.026940798s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:08.284021691 (720h46m53.746114164s) [upload=634034]: Next returned nil segment Oct 23 14:18:08.284044505 (720h46m53.746136978s) [upload=634034]: Waiting for error group managing segments... Oct 23 14:18:08.566598218 (720h46m54.028690691s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.594453381 (720h46m54.056545864s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.598953419 (720h46m54.061045892s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.602023124 (720h46m54.064115658s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625098415 (720h46m54.087190898s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625760095 (720h46m54.087852578s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.631004962 (720h46m54.093097445s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.635035678 (720h46m54.097128161s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.645903372 (720h46m54.107995845s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.711835221 (720h46m54.173927704s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.719309664 (720h46m54.181402147s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.758298877 (720h46m54.22039136s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819795110 (720h46m54.281887593s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819860023 (720h46m54.281952516s) [seg_pos={0 0}, upload=634034]: Segment reached optimal threshold of 13 pieces. Oct 23 14:18:08.820128314 (720h46m54.282220807s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection Oct 23 14:18:08.820140978 (720h46m54.282233461s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820198287 (720h46m54.28229077s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820209729 (720h46m54.282302222s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820445267 (720h46m54.28253775s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820509670 (720h46m54.282602153s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820531111 (720h46m54.282623594s) [seg_pos={0 0}, upload=634034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:18:08.820624719 (720h46m54.282717202s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:18:08.820630982 (720h46m54.282723465s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:18:09.121056837 (720h46m54.58314933s) [upload=634034]: Done uploading segments... Oct 23 14:18:09.121104197 (720h46m54.58319668s) [upload=634034]: Upload finished. err: <nil> Duration: 5.839430687s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.077508991s FAIL (2025-10-23 14:35:23 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 2.077549097s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 6.975124955s FAIL (2025-10-23 14:35:03 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.975155412s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.05440552s FAIL (2025-10-23 14:34:58 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 2.054425527s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X.........X......................] duration: 5.984342968s FAIL (2025-10-23 14:34:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:18:06.304157422 (720h46m51.766249905s) [upload=634034]: Starting upload Oct 23 14:18:06.304347725 (720h46m51.766440218s) [upload=634034]: Done starting upload Oct 23 14:18:06.304632076 (720h46m51.766724559s) [upload=634034]: Uploading segments... Oct 23 14:18:06.304690778 (720h46m51.766783261s) [upload=634034]: Got next segment. Inline: false Oct 23 14:18:06.304730654 (720h46m51.766823147s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:18:06.304737897 (720h46m51.766830371s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:18:06.560110142 (720h46m52.022202615s) [seg_pos={0 0}, upload=634034]: Begin upload segment... Oct 23 14:18:06.560170466 (720h46m52.022262949s) [seg_pos={0 0}, upload=634034]: Done begin upload segment. Oct 23 14:18:06.560196877 (720h46m52.02228936s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.560384274 (720h46m52.022476767s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561087734 (720h46m52.023180217s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561246465 (720h46m52.023338948s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561388856 (720h46m52.02348134s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561682246 (720h46m52.023774729s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561840206 (720h46m52.023932699s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562036520 (720h46m52.024129003s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562369515 (720h46m52.024461998s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562402808 (720h46m52.024495301s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562804713 (720h46m52.024897196s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562996849 (720h46m52.025089332s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563336766 (720h46m52.025429249s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563574799 (720h46m52.025667282s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564175863 (720h46m52.026268346s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564848324 (720h46m52.026940798s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:08.284021691 (720h46m53.746114164s) [upload=634034]: Next returned nil segment Oct 23 14:18:08.284044505 (720h46m53.746136978s) [upload=634034]: Waiting for error group managing segments... Oct 23 14:18:08.566598218 (720h46m54.028690691s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.594453381 (720h46m54.056545864s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.598953419 (720h46m54.061045892s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.602023124 (720h46m54.064115658s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625098415 (720h46m54.087190898s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625760095 (720h46m54.087852578s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.631004962 (720h46m54.093097445s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.635035678 (720h46m54.097128161s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.645903372 (720h46m54.107995845s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.711835221 (720h46m54.173927704s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.719309664 (720h46m54.181402147s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.758298877 (720h46m54.22039136s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819795110 (720h46m54.281887593s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819860023 (720h46m54.281952516s) [seg_pos={0 0}, upload=634034]: Segment reached optimal threshold of 13 pieces. Oct 23 14:18:08.820128314 (720h46m54.282220807s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection Oct 23 14:18:08.820140978 (720h46m54.282233461s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820198287 (720h46m54.28229077s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820209729 (720h46m54.282302222s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820445267 (720h46m54.28253775s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820509670 (720h46m54.282602153s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820531111 (720h46m54.282623594s) [seg_pos={0 0}, upload=634034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:18:08.820624719 (720h46m54.282717202s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:18:08.820630982 (720h46m54.282723465s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:18:09.121056837 (720h46m54.58314933s) [upload=634034]: Done uploading segments... Oct 23 14:18:09.121104197 (720h46m54.58319668s) [upload=634034]: Upload finished. err: <nil> Duration: 5.984371953s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.031003987s FAIL (2025-10-23 14:34:08 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 2.031037882s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 6.340950625s FAIL (2025-10-23 14:34:06 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.34099481s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.078467088s FAIL (2025-10-23 14:33:46 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 2.078498918s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X.........X.....................] duration: 5.889457998s FAIL (2025-10-23 14:33:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:18:06.304157422 (720h46m51.766249905s) [upload=634034]: Starting upload Oct 23 14:18:06.304347725 (720h46m51.766440218s) [upload=634034]: Done starting upload Oct 23 14:18:06.304632076 (720h46m51.766724559s) [upload=634034]: Uploading segments... Oct 23 14:18:06.304690778 (720h46m51.766783261s) [upload=634034]: Got next segment. Inline: false Oct 23 14:18:06.304730654 (720h46m51.766823147s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:18:06.304737897 (720h46m51.766830371s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:18:06.560110142 (720h46m52.022202615s) [seg_pos={0 0}, upload=634034]: Begin upload segment... Oct 23 14:18:06.560170466 (720h46m52.022262949s) [seg_pos={0 0}, upload=634034]: Done begin upload segment. Oct 23 14:18:06.560196877 (720h46m52.02228936s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.560384274 (720h46m52.022476767s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561087734 (720h46m52.023180217s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561246465 (720h46m52.023338948s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561388856 (720h46m52.02348134s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561682246 (720h46m52.023774729s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561840206 (720h46m52.023932699s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562036520 (720h46m52.024129003s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562369515 (720h46m52.024461998s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562402808 (720h46m52.024495301s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562804713 (720h46m52.024897196s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562996849 (720h46m52.025089332s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563336766 (720h46m52.025429249s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563574799 (720h46m52.025667282s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564175863 (720h46m52.026268346s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564848324 (720h46m52.026940798s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:08.284021691 (720h46m53.746114164s) [upload=634034]: Next returned nil segment Oct 23 14:18:08.284044505 (720h46m53.746136978s) [upload=634034]: Waiting for error group managing segments... Oct 23 14:18:08.566598218 (720h46m54.028690691s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.594453381 (720h46m54.056545864s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.598953419 (720h46m54.061045892s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.602023124 (720h46m54.064115658s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625098415 (720h46m54.087190898s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625760095 (720h46m54.087852578s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.631004962 (720h46m54.093097445s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.635035678 (720h46m54.097128161s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.645903372 (720h46m54.107995845s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.711835221 (720h46m54.173927704s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.719309664 (720h46m54.181402147s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.758298877 (720h46m54.22039136s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819795110 (720h46m54.281887593s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819860023 (720h46m54.281952516s) [seg_pos={0 0}, upload=634034]: Segment reached optimal threshold of 13 pieces. Oct 23 14:18:08.820128314 (720h46m54.282220807s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection Oct 23 14:18:08.820140978 (720h46m54.282233461s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820198287 (720h46m54.28229077s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820209729 (720h46m54.282302222s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820445267 (720h46m54.28253775s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820509670 (720h46m54.282602153s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820531111 (720h46m54.282623594s) [seg_pos={0 0}, upload=634034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:18:08.820624719 (720h46m54.282717202s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:18:08.820630982 (720h46m54.282723465s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:18:09.121056837 (720h46m54.58314933s) [upload=634034]: Done uploading segments... Oct 23 14:18:09.121104197 (720h46m54.58319668s) [upload=634034]: Upload finished. err: <nil> Duration: 5.889489169s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 6.028687311s FAIL (2025-10-23 14:33:35 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.028722799s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.052713988s FAIL (2025-10-23 14:33:27 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 2.052746288s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 6.55627114s FAIL (2025-10-23 14:33:09 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.556298873s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X.........X....................] duration: 5.890223597s FAIL (2025-10-23 14:32:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:18:06.304157422 (720h46m51.766249905s) [upload=634034]: Starting upload Oct 23 14:18:06.304347725 (720h46m51.766440218s) [upload=634034]: Done starting upload Oct 23 14:18:06.304632076 (720h46m51.766724559s) [upload=634034]: Uploading segments... Oct 23 14:18:06.304690778 (720h46m51.766783261s) [upload=634034]: Got next segment. Inline: false Oct 23 14:18:06.304730654 (720h46m51.766823147s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:18:06.304737897 (720h46m51.766830371s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:18:06.560110142 (720h46m52.022202615s) [seg_pos={0 0}, upload=634034]: Begin upload segment... Oct 23 14:18:06.560170466 (720h46m52.022262949s) [seg_pos={0 0}, upload=634034]: Done begin upload segment. Oct 23 14:18:06.560196877 (720h46m52.02228936s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.560384274 (720h46m52.022476767s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561087734 (720h46m52.023180217s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561246465 (720h46m52.023338948s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561388856 (720h46m52.02348134s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561682246 (720h46m52.023774729s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561840206 (720h46m52.023932699s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562036520 (720h46m52.024129003s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562369515 (720h46m52.024461998s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562402808 (720h46m52.024495301s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562804713 (720h46m52.024897196s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562996849 (720h46m52.025089332s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563336766 (720h46m52.025429249s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563574799 (720h46m52.025667282s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564175863 (720h46m52.026268346s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564848324 (720h46m52.026940798s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:08.284021691 (720h46m53.746114164s) [upload=634034]: Next returned nil segment Oct 23 14:18:08.284044505 (720h46m53.746136978s) [upload=634034]: Waiting for error group managing segments... Oct 23 14:18:08.566598218 (720h46m54.028690691s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.594453381 (720h46m54.056545864s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.598953419 (720h46m54.061045892s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.602023124 (720h46m54.064115658s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625098415 (720h46m54.087190898s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625760095 (720h46m54.087852578s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.631004962 (720h46m54.093097445s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.635035678 (720h46m54.097128161s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.645903372 (720h46m54.107995845s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.711835221 (720h46m54.173927704s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.719309664 (720h46m54.181402147s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.758298877 (720h46m54.22039136s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819795110 (720h46m54.281887593s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819860023 (720h46m54.281952516s) [seg_pos={0 0}, upload=634034]: Segment reached optimal threshold of 13 pieces. Oct 23 14:18:08.820128314 (720h46m54.282220807s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection Oct 23 14:18:08.820140978 (720h46m54.282233461s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820198287 (720h46m54.28229077s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820209729 (720h46m54.282302222s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820445267 (720h46m54.28253775s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820509670 (720h46m54.282602153s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820531111 (720h46m54.282623594s) [seg_pos={0 0}, upload=634034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:18:08.820624719 (720h46m54.282717202s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:18:08.820630982 (720h46m54.282723465s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:18:09.121056837 (720h46m54.58314933s) [upload=634034]: Done uploading segments... Oct 23 14:18:09.121104197 (720h46m54.58319668s) [upload=634034]: Upload finished. err: <nil> Duration: 5.890244396s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.016000378s FAIL (2025-10-23 14:32:38 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 2.016049442s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 6.938844027s FAIL (2025-10-23 14:32:34 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.938879866s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.760540907s FAIL (2025-10-23 14:32:01 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.760597965s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X.........X...................] duration: 5.848584526s FAIL (2025-10-23 14:31:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:18:06.304157422 (720h46m51.766249905s) [upload=634034]: Starting upload Oct 23 14:18:06.304347725 (720h46m51.766440218s) [upload=634034]: Done starting upload Oct 23 14:18:06.304632076 (720h46m51.766724559s) [upload=634034]: Uploading segments... Oct 23 14:18:06.304690778 (720h46m51.766783261s) [upload=634034]: Got next segment. Inline: false Oct 23 14:18:06.304730654 (720h46m51.766823147s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:18:06.304737897 (720h46m51.766830371s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:18:06.560110142 (720h46m52.022202615s) [seg_pos={0 0}, upload=634034]: Begin upload segment... Oct 23 14:18:06.560170466 (720h46m52.022262949s) [seg_pos={0 0}, upload=634034]: Done begin upload segment. Oct 23 14:18:06.560196877 (720h46m52.02228936s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.560384274 (720h46m52.022476767s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561087734 (720h46m52.023180217s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561246465 (720h46m52.023338948s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561388856 (720h46m52.02348134s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561682246 (720h46m52.023774729s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561840206 (720h46m52.023932699s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562036520 (720h46m52.024129003s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562369515 (720h46m52.024461998s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562402808 (720h46m52.024495301s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562804713 (720h46m52.024897196s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562996849 (720h46m52.025089332s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563336766 (720h46m52.025429249s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563574799 (720h46m52.025667282s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564175863 (720h46m52.026268346s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564848324 (720h46m52.026940798s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:08.284021691 (720h46m53.746114164s) [upload=634034]: Next returned nil segment Oct 23 14:18:08.284044505 (720h46m53.746136978s) [upload=634034]: Waiting for error group managing segments... Oct 23 14:18:08.566598218 (720h46m54.028690691s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.594453381 (720h46m54.056545864s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.598953419 (720h46m54.061045892s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.602023124 (720h46m54.064115658s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625098415 (720h46m54.087190898s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625760095 (720h46m54.087852578s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.631004962 (720h46m54.093097445s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.635035678 (720h46m54.097128161s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.645903372 (720h46m54.107995845s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.711835221 (720h46m54.173927704s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.719309664 (720h46m54.181402147s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.758298877 (720h46m54.22039136s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819795110 (720h46m54.281887593s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819860023 (720h46m54.281952516s) [seg_pos={0 0}, upload=634034]: Segment reached optimal threshold of 13 pieces. Oct 23 14:18:08.820128314 (720h46m54.282220807s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection Oct 23 14:18:08.820140978 (720h46m54.282233461s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820198287 (720h46m54.28229077s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820209729 (720h46m54.282302222s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820445267 (720h46m54.28253775s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820509670 (720h46m54.282602153s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820531111 (720h46m54.282623594s) [seg_pos={0 0}, upload=634034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:18:08.820624719 (720h46m54.282717202s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:18:08.820630982 (720h46m54.282723465s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:18:09.121056837 (720h46m54.58314933s) [upload=634034]: Done uploading segments... Oct 23 14:18:09.121104197 (720h46m54.58319668s) [upload=634034]: Upload finished. err: <nil> Duration: 5.8486188s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 7.326862454s FAIL (2025-10-23 14:31:44 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 7.326898935s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.104261778s FAIL (2025-10-23 14:31:33 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 2.104302266s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X.........X..................] duration: 5.970292905s FAIL (2025-10-23 14:31:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:18:06.304157422 (720h46m51.766249905s) [upload=634034]: Starting upload Oct 23 14:18:06.304347725 (720h46m51.766440218s) [upload=634034]: Done starting upload Oct 23 14:18:06.304632076 (720h46m51.766724559s) [upload=634034]: Uploading segments... Oct 23 14:18:06.304690778 (720h46m51.766783261s) [upload=634034]: Got next segment. Inline: false Oct 23 14:18:06.304730654 (720h46m51.766823147s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:18:06.304737897 (720h46m51.766830371s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:18:06.560110142 (720h46m52.022202615s) [seg_pos={0 0}, upload=634034]: Begin upload segment... Oct 23 14:18:06.560170466 (720h46m52.022262949s) [seg_pos={0 0}, upload=634034]: Done begin upload segment. Oct 23 14:18:06.560196877 (720h46m52.02228936s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.560384274 (720h46m52.022476767s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561087734 (720h46m52.023180217s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561246465 (720h46m52.023338948s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561388856 (720h46m52.02348134s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561682246 (720h46m52.023774729s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561840206 (720h46m52.023932699s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562036520 (720h46m52.024129003s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562369515 (720h46m52.024461998s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562402808 (720h46m52.024495301s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562804713 (720h46m52.024897196s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562996849 (720h46m52.025089332s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563336766 (720h46m52.025429249s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563574799 (720h46m52.025667282s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564175863 (720h46m52.026268346s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564848324 (720h46m52.026940798s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:08.284021691 (720h46m53.746114164s) [upload=634034]: Next returned nil segment Oct 23 14:18:08.284044505 (720h46m53.746136978s) [upload=634034]: Waiting for error group managing segments... Oct 23 14:18:08.566598218 (720h46m54.028690691s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.594453381 (720h46m54.056545864s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.598953419 (720h46m54.061045892s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.602023124 (720h46m54.064115658s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625098415 (720h46m54.087190898s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625760095 (720h46m54.087852578s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.631004962 (720h46m54.093097445s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.635035678 (720h46m54.097128161s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.645903372 (720h46m54.107995845s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.711835221 (720h46m54.173927704s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.719309664 (720h46m54.181402147s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.758298877 (720h46m54.22039136s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819795110 (720h46m54.281887593s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819860023 (720h46m54.281952516s) [seg_pos={0 0}, upload=634034]: Segment reached optimal threshold of 13 pieces. Oct 23 14:18:08.820128314 (720h46m54.282220807s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection Oct 23 14:18:08.820140978 (720h46m54.282233461s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820198287 (720h46m54.28229077s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820209729 (720h46m54.282302222s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820445267 (720h46m54.28253775s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820509670 (720h46m54.282602153s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820531111 (720h46m54.282623594s) [seg_pos={0 0}, upload=634034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:18:08.820624719 (720h46m54.282717202s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:18:08.820630982 (720h46m54.282723465s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:18:09.121056837 (720h46m54.58314933s) [upload=634034]: Done uploading segments... Oct 23 14:18:09.121104197 (720h46m54.58319668s) [upload=634034]: Upload finished. err: <nil> Duration: 5.970317042s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 8.64031286s FAIL (2025-10-23 14:31:16 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 8.64034474s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X.........X.................] duration: 6.042575849s FAIL (2025-10-23 14:30:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:18:06.304157422 (720h46m51.766249905s) [upload=634034]: Starting upload Oct 23 14:18:06.304347725 (720h46m51.766440218s) [upload=634034]: Done starting upload Oct 23 14:18:06.304632076 (720h46m51.766724559s) [upload=634034]: Uploading segments... Oct 23 14:18:06.304690778 (720h46m51.766783261s) [upload=634034]: Got next segment. Inline: false Oct 23 14:18:06.304730654 (720h46m51.766823147s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:18:06.304737897 (720h46m51.766830371s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:18:06.560110142 (720h46m52.022202615s) [seg_pos={0 0}, upload=634034]: Begin upload segment... Oct 23 14:18:06.560170466 (720h46m52.022262949s) [seg_pos={0 0}, upload=634034]: Done begin upload segment. Oct 23 14:18:06.560196877 (720h46m52.02228936s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.560384274 (720h46m52.022476767s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561087734 (720h46m52.023180217s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561246465 (720h46m52.023338948s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561388856 (720h46m52.02348134s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561682246 (720h46m52.023774729s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561840206 (720h46m52.023932699s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562036520 (720h46m52.024129003s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562369515 (720h46m52.024461998s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562402808 (720h46m52.024495301s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562804713 (720h46m52.024897196s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562996849 (720h46m52.025089332s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563336766 (720h46m52.025429249s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563574799 (720h46m52.025667282s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564175863 (720h46m52.026268346s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564848324 (720h46m52.026940798s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:08.284021691 (720h46m53.746114164s) [upload=634034]: Next returned nil segment Oct 23 14:18:08.284044505 (720h46m53.746136978s) [upload=634034]: Waiting for error group managing segments... Oct 23 14:18:08.566598218 (720h46m54.028690691s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.594453381 (720h46m54.056545864s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.598953419 (720h46m54.061045892s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.602023124 (720h46m54.064115658s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625098415 (720h46m54.087190898s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625760095 (720h46m54.087852578s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.631004962 (720h46m54.093097445s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.635035678 (720h46m54.097128161s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.645903372 (720h46m54.107995845s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.711835221 (720h46m54.173927704s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.719309664 (720h46m54.181402147s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.758298877 (720h46m54.22039136s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819795110 (720h46m54.281887593s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819860023 (720h46m54.281952516s) [seg_pos={0 0}, upload=634034]: Segment reached optimal threshold of 13 pieces. Oct 23 14:18:08.820128314 (720h46m54.282220807s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection Oct 23 14:18:08.820140978 (720h46m54.282233461s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820198287 (720h46m54.28229077s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820209729 (720h46m54.282302222s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820445267 (720h46m54.28253775s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820509670 (720h46m54.282602153s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820531111 (720h46m54.282623594s) [seg_pos={0 0}, upload=634034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:18:08.820624719 (720h46m54.282717202s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:18:08.820630982 (720h46m54.282723465s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:18:09.121056837 (720h46m54.58314933s) [upload=634034]: Done uploading segments... Oct 23 14:18:09.121104197 (720h46m54.58319668s) [upload=634034]: Upload finished. err: <nil> Duration: 6.042600066s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 9.189496258s FAIL (2025-10-23 14:30:43 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 9.18953424s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 3.833079752s FAIL (2025-10-23 14:30:22 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 3.833116213s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 6.697826017s FAIL (2025-10-23 14:30:13 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.697865592s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X.........X................] duration: 8.993327526s FAIL (2025-10-23 14:29:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:18:06.304157422 (720h46m51.766249905s) [upload=634034]: Starting upload Oct 23 14:18:06.304347725 (720h46m51.766440218s) [upload=634034]: Done starting upload Oct 23 14:18:06.304632076 (720h46m51.766724559s) [upload=634034]: Uploading segments... Oct 23 14:18:06.304690778 (720h46m51.766783261s) [upload=634034]: Got next segment. Inline: false Oct 23 14:18:06.304730654 (720h46m51.766823147s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:18:06.304737897 (720h46m51.766830371s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:18:06.560110142 (720h46m52.022202615s) [seg_pos={0 0}, upload=634034]: Begin upload segment... Oct 23 14:18:06.560170466 (720h46m52.022262949s) [seg_pos={0 0}, upload=634034]: Done begin upload segment. Oct 23 14:18:06.560196877 (720h46m52.02228936s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.560384274 (720h46m52.022476767s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561087734 (720h46m52.023180217s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561246465 (720h46m52.023338948s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561388856 (720h46m52.02348134s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561682246 (720h46m52.023774729s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561840206 (720h46m52.023932699s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562036520 (720h46m52.024129003s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562369515 (720h46m52.024461998s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562402808 (720h46m52.024495301s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562804713 (720h46m52.024897196s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562996849 (720h46m52.025089332s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563336766 (720h46m52.025429249s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563574799 (720h46m52.025667282s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564175863 (720h46m52.026268346s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564848324 (720h46m52.026940798s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:08.284021691 (720h46m53.746114164s) [upload=634034]: Next returned nil segment Oct 23 14:18:08.284044505 (720h46m53.746136978s) [upload=634034]: Waiting for error group managing segments... Oct 23 14:18:08.566598218 (720h46m54.028690691s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.594453381 (720h46m54.056545864s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.598953419 (720h46m54.061045892s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.602023124 (720h46m54.064115658s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625098415 (720h46m54.087190898s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625760095 (720h46m54.087852578s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.631004962 (720h46m54.093097445s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.635035678 (720h46m54.097128161s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.645903372 (720h46m54.107995845s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.711835221 (720h46m54.173927704s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.719309664 (720h46m54.181402147s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.758298877 (720h46m54.22039136s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819795110 (720h46m54.281887593s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819860023 (720h46m54.281952516s) [seg_pos={0 0}, upload=634034]: Segment reached optimal threshold of 13 pieces. Oct 23 14:18:08.820128314 (720h46m54.282220807s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection Oct 23 14:18:08.820140978 (720h46m54.282233461s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820198287 (720h46m54.28229077s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820209729 (720h46m54.282302222s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820445267 (720h46m54.28253775s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820509670 (720h46m54.282602153s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820531111 (720h46m54.282623594s) [seg_pos={0 0}, upload=634034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:18:08.820624719 (720h46m54.282717202s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:18:08.820630982 (720h46m54.282723465s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:18:09.121056837 (720h46m54.58314933s) [upload=634034]: Done uploading segments... Oct 23 14:18:09.121104197 (720h46m54.58319668s) [upload=634034]: Upload finished. err: <nil> Duration: 8.993362331s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.194066226s FAIL (2025-10-23 14:29:45 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 2.194105861s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 7.62246728s FAIL (2025-10-23 14:29:31 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 7.62249904s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 11.87745696s FAIL (2025-10-23 14:29:18 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 11.877489291s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 3.285021642s FAIL (2025-10-23 14:29:11 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 3.285058281s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X.........X...............] duration: 6.317451942s FAIL (2025-10-23 14:28:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:18:06.304157422 (720h46m51.766249905s) [upload=634034]: Starting upload Oct 23 14:18:06.304347725 (720h46m51.766440218s) [upload=634034]: Done starting upload Oct 23 14:18:06.304632076 (720h46m51.766724559s) [upload=634034]: Uploading segments... Oct 23 14:18:06.304690778 (720h46m51.766783261s) [upload=634034]: Got next segment. Inline: false Oct 23 14:18:06.304730654 (720h46m51.766823147s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:18:06.304737897 (720h46m51.766830371s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:18:06.560110142 (720h46m52.022202615s) [seg_pos={0 0}, upload=634034]: Begin upload segment... Oct 23 14:18:06.560170466 (720h46m52.022262949s) [seg_pos={0 0}, upload=634034]: Done begin upload segment. Oct 23 14:18:06.560196877 (720h46m52.02228936s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.560384274 (720h46m52.022476767s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561087734 (720h46m52.023180217s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561246465 (720h46m52.023338948s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561388856 (720h46m52.02348134s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561682246 (720h46m52.023774729s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561840206 (720h46m52.023932699s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562036520 (720h46m52.024129003s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562369515 (720h46m52.024461998s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562402808 (720h46m52.024495301s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562804713 (720h46m52.024897196s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562996849 (720h46m52.025089332s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563336766 (720h46m52.025429249s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563574799 (720h46m52.025667282s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564175863 (720h46m52.026268346s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564848324 (720h46m52.026940798s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:08.284021691 (720h46m53.746114164s) [upload=634034]: Next returned nil segment Oct 23 14:18:08.284044505 (720h46m53.746136978s) [upload=634034]: Waiting for error group managing segments... Oct 23 14:18:08.566598218 (720h46m54.028690691s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.594453381 (720h46m54.056545864s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.598953419 (720h46m54.061045892s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.602023124 (720h46m54.064115658s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625098415 (720h46m54.087190898s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625760095 (720h46m54.087852578s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.631004962 (720h46m54.093097445s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.635035678 (720h46m54.097128161s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.645903372 (720h46m54.107995845s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.711835221 (720h46m54.173927704s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.719309664 (720h46m54.181402147s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.758298877 (720h46m54.22039136s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819795110 (720h46m54.281887593s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819860023 (720h46m54.281952516s) [seg_pos={0 0}, upload=634034]: Segment reached optimal threshold of 13 pieces. Oct 23 14:18:08.820128314 (720h46m54.282220807s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection Oct 23 14:18:08.820140978 (720h46m54.282233461s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820198287 (720h46m54.28229077s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820209729 (720h46m54.282302222s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820445267 (720h46m54.28253775s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820509670 (720h46m54.282602153s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820531111 (720h46m54.282623594s) [seg_pos={0 0}, upload=634034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:18:08.820624719 (720h46m54.282717202s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:18:08.820630982 (720h46m54.282723465s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:18:09.121056837 (720h46m54.58314933s) [upload=634034]: Done uploading segments... Oct 23 14:18:09.121104197 (720h46m54.58319668s) [upload=634034]: Upload finished. err: <nil> Duration: 6.317484142s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X.........X..............] duration: 6.131535607s FAIL (2025-10-23 14:28:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:18:06.304157422 (720h46m51.766249905s) [upload=634034]: Starting upload Oct 23 14:18:06.304347725 (720h46m51.766440218s) [upload=634034]: Done starting upload Oct 23 14:18:06.304632076 (720h46m51.766724559s) [upload=634034]: Uploading segments... Oct 23 14:18:06.304690778 (720h46m51.766783261s) [upload=634034]: Got next segment. Inline: false Oct 23 14:18:06.304730654 (720h46m51.766823147s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:18:06.304737897 (720h46m51.766830371s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:18:06.560110142 (720h46m52.022202615s) [seg_pos={0 0}, upload=634034]: Begin upload segment... Oct 23 14:18:06.560170466 (720h46m52.022262949s) [seg_pos={0 0}, upload=634034]: Done begin upload segment. Oct 23 14:18:06.560196877 (720h46m52.02228936s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.560384274 (720h46m52.022476767s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561087734 (720h46m52.023180217s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561246465 (720h46m52.023338948s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561388856 (720h46m52.02348134s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561682246 (720h46m52.023774729s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561840206 (720h46m52.023932699s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562036520 (720h46m52.024129003s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562369515 (720h46m52.024461998s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562402808 (720h46m52.024495301s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562804713 (720h46m52.024897196s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562996849 (720h46m52.025089332s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563336766 (720h46m52.025429249s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563574799 (720h46m52.025667282s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564175863 (720h46m52.026268346s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564848324 (720h46m52.026940798s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:08.284021691 (720h46m53.746114164s) [upload=634034]: Next returned nil segment Oct 23 14:18:08.284044505 (720h46m53.746136978s) [upload=634034]: Waiting for error group managing segments... Oct 23 14:18:08.566598218 (720h46m54.028690691s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.594453381 (720h46m54.056545864s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.598953419 (720h46m54.061045892s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.602023124 (720h46m54.064115658s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625098415 (720h46m54.087190898s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625760095 (720h46m54.087852578s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.631004962 (720h46m54.093097445s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.635035678 (720h46m54.097128161s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.645903372 (720h46m54.107995845s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.711835221 (720h46m54.173927704s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.719309664 (720h46m54.181402147s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.758298877 (720h46m54.22039136s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819795110 (720h46m54.281887593s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819860023 (720h46m54.281952516s) [seg_pos={0 0}, upload=634034]: Segment reached optimal threshold of 13 pieces. Oct 23 14:18:08.820128314 (720h46m54.282220807s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection Oct 23 14:18:08.820140978 (720h46m54.282233461s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820198287 (720h46m54.28229077s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820209729 (720h46m54.282302222s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820445267 (720h46m54.28253775s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820509670 (720h46m54.282602153s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820531111 (720h46m54.282623594s) [seg_pos={0 0}, upload=634034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:18:08.820624719 (720h46m54.282717202s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:18:08.820630982 (720h46m54.282723465s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:18:09.121056837 (720h46m54.58314933s) [upload=634034]: Done uploading segments... Oct 23 14:18:09.121104197 (720h46m54.58319668s) [upload=634034]: Upload finished. err: <nil> Duration: 6.131567346s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.559055897s FAIL (2025-10-23 14:28:27 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 2.559087468s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.694421939s FAIL (2025-10-23 14:28:09 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.694451095s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 4.991581055s FAIL (2025-10-23 14:27:43 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 4.991603307s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X.........X.............] duration: 7.271287528s FAIL (2025-10-23 14:27:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 23 14:18:06.304157422 (720h46m51.766249905s) [upload=634034]: Starting upload Oct 23 14:18:06.304347725 (720h46m51.766440218s) [upload=634034]: Done starting upload Oct 23 14:18:06.304632076 (720h46m51.766724559s) [upload=634034]: Uploading segments... Oct 23 14:18:06.304690778 (720h46m51.766783261s) [upload=634034]: Got next segment. Inline: false Oct 23 14:18:06.304730654 (720h46m51.766823147s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 23 14:18:06.304737897 (720h46m51.766830371s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 23 14:18:06.560110142 (720h46m52.022202615s) [seg_pos={0 0}, upload=634034]: Begin upload segment... Oct 23 14:18:06.560170466 (720h46m52.022262949s) [seg_pos={0 0}, upload=634034]: Done begin upload segment. Oct 23 14:18:06.560196877 (720h46m52.02228936s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.560384274 (720h46m52.022476767s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561087734 (720h46m52.023180217s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561246465 (720h46m52.023338948s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561388856 (720h46m52.02348134s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561682246 (720h46m52.023774729s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.561840206 (720h46m52.023932699s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562036520 (720h46m52.024129003s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562369515 (720h46m52.024461998s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562402808 (720h46m52.024495301s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562804713 (720h46m52.024897196s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.562996849 (720h46m52.025089332s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563336766 (720h46m52.025429249s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.563574799 (720h46m52.025667282s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564175863 (720h46m52.026268346s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:06.564848324 (720h46m52.026940798s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Uploading piece... Oct 23 14:18:08.284021691 (720h46m53.746114164s) [upload=634034]: Next returned nil segment Oct 23 14:18:08.284044505 (720h46m53.746136978s) [upload=634034]: Waiting for error group managing segments... Oct 23 14:18:08.566598218 (720h46m54.028690691s) [noise=true , address=145.79.227.3:4151 , piece_id=UKN722E3IFNUMWPO52OOYWIMZBNBNY7ABKQP5QMTMEZPEYERISNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.594453381 (720h46m54.056545864s) [noise=true , address=145.79.227.7:4293 , piece_id=VUDI2LJ2V2AEBCLGSCYLGRJABRNTY5NCAKRKPLZCXNWZEJS7J3JA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.598953419 (720h46m54.061045892s) [noise=true , address=145.79.227.18:4006 , piece_id=D2M333DOFU7OK72FNM2V72R3ROQOG67A7TNB2WLUZEGVPFXDKHNQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.602023124 (720h46m54.064115658s) [noise=true , address=145.79.227.24:4583 , piece_id=QUOAUGFR3QHS2VOM4B5CKPDYAARBCJQO4YLQ6D4ZCZEXBONIWTUA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625098415 (720h46m54.087190898s) [noise=true , address=145.79.227.4:4185 , piece_id=ST3KYCA2RMGKFDSEAAME7AHAJVYUDRBBGD5FX5H3D3Q2HWI3YXXA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.625760095 (720h46m54.087852578s) [noise=true , address=145.79.227.22:4511 , piece_id=BZYS462ZPMZJ7SSFUL3DLZVC545VYECFOTJLZ2KXKMXPPWNNRL4Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.631004962 (720h46m54.093097445s) [noise=true , address=145.79.227.8:4331 , piece_id=ZOEWWVA73T53XVXKZWGBWSCGTPFQTPLA6WLOO3VE2X3B2HEPQSMQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.635035678 (720h46m54.097128161s) [noise=true , address=145.79.227.26:4653 , piece_id=DHFMJOP4YLMJX454M7H27QE4TRZ66YYOXOT2USABU6U6SEHHP4AA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.645903372 (720h46m54.107995845s) [noise=true , address=145.79.227.3:4150 , piece_id=JN2GSP4PFLSSGYHHAGCLA26FPD6YDX22ZTFWBJWGL6TCAX3RB6KQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.711835221 (720h46m54.173927704s) [noise=true , address=145.79.227.6:4258 , piece_id=7FKU4V3O62ZV4XZFLULPBBJOMRY33UORR3GR4N5LGJY7JYXU4I5Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.719309664 (720h46m54.181402147s) [noise=true , address=145.79.227.19:4077 , piece_id=NKCLT6JST3XSQ6YAK2GJSS3WUTMNTC57NW5VSZTADPRL7COAIK3Q, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.758298877 (720h46m54.22039136s) [noise=true , address=45.91.116.6:4975 , piece_id=63KIP6GK2TXBGQ6CW7PQH2RJJ4TRWRZBHLFWGDVV3JUV6YKZQ4YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819795110 (720h46m54.281887593s) [noise=true , address=77.90.31.14:4661 , piece_id=4P7H6P3HCNEXOTARMZKGZHMZ5DKEX4NRLX4DJIBMB57IEKNT4BWA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: <nil> Oct 23 14:18:08.819860023 (720h46m54.281952516s) [seg_pos={0 0}, upload=634034]: Segment reached optimal threshold of 13 pieces. Oct 23 14:18:08.820128314 (720h46m54.282220807s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9): protocol: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41906->77.90.31.16:4734: use of closed network connection Oct 23 14:18:08.820140978 (720h46m54.282233461s) [noise=true , address=77.90.31.16:4734 , piece_id=7FD3GHKFZOPBJ26WB6LTCIOAL5IPMSGS3BQBJZ3ANMOI3GRX4RLQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820198287 (720h46m54.28229077s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820209729 (720h46m54.282302222s) [noise=true , address=77.90.31.18:4806 , piece_id=D34LR2JBKSXUYODPZ72WOUUP2KKVERRRIBWJXV2EC7ESKMTAG3HA, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820445267 (720h46m54.28253775s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Oct 23 14:18:08.820509670 (720h46m54.282602153s) [noise=true , address=77.90.31.11:4554 , piece_id=O7QXIK4QMZDITNFWNNNQRJXVXJFHNAUOIGUGW5LEGI7HBX6WX3YQ, seg_pos={0 0}, upload=634034]: Long tail cancellation due to optimal threshold Oct 23 14:18:08.820531111 (720h46m54.282623594s) [seg_pos={0 0}, upload=634034]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 23 14:18:08.820624719 (720h46m54.282717202s) [upload=634034]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 23 14:18:08.820630982 (720h46m54.282723465s) [upload=634034]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 23 14:18:09.121056837 (720h46m54.58314933s) [upload=634034]: Done uploading segments... Oct 23 14:18:09.121104197 (720h46m54.58319668s) [upload=634034]: Upload finished. err: <nil> Duration: 7.271311494s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 8.346306856s FAIL (2025-10-23 14:27:06 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 8.346341041s core uplink us1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)