Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X........X.....................XXXX...X..X...........X......] duration: 6.150969076s trace: https://jaeger.dev.storj.io/trace/fa7962f52d9630d FAIL (2025-12-21 13:29:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: MN5IF2YLUACGZTO4BG2UQWBCD3MLHDBGHYSHO5PUYVVLODSL7H5A) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: JKUFZKN4XVAW7BIVYPOQJURRNEG7I444RJSZC5IGH2X5FFQOJ2OA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: L72GMUZ6FNO3ATO3OT2H2EZEUM3NATEGRKBSHUSGURSGETRV3BSA) context deadline exceeded ======LOGS====== gments... Dec 21 13:24:53.199397540 (381h55m53.070480575s) [upload=335186]: Got next segment. Inline: false Dec 21 13:24:53.199433087 (381h55m53.070516122s) [upload=335186]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 13:24:53.199437856 (381h55m53.070520901s) [upload=335186]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 13:24:53.446838205 (381h55m53.31792125s) [seg_pos={0 0}, upload=335186]: Begin upload segment... Dec 21 13:24:53.446894843 (381h55m53.317977878s) [seg_pos={0 0}, upload=335186]: Done begin upload segment. Dec 21 13:24:53.446931453 (381h55m53.318014488s) [noise=true , address=145.79.227.51:4409 , piece_id=F2B7K7BE7JLBZMIHP5STKVAN7MH2PCYKJ33ZYJF4XJMABZSI75YA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.446966390 (381h55m53.318049435s) [noise=true , address=145.79.227.43:4157 , piece_id=CUGHWACL4LKBK3K5R6AYAU7PZSWTR5LH2KDP6JVV36XMFJVAQRAA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.447612399 (381h55m53.318695444s) [noise=true , address=145.79.227.44:4195 , piece_id=QECQLX7KZVU7RMFVZ6TOHRIXR776DFZ4D44FKFNUWYH2EURRYX7Q, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.447623340 (381h55m53.318706395s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4CVCVL6PSTVUM33DZPLIU6QUN6JIXFCPBU6CNNA4P6YCHJ3PONA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.448102392 (381h55m53.319185437s) [noise=true , address=145.79.227.35:4481 , piece_id=BBT67CINYVQGAHO5U6KGRG6VDPVKUFQXMWWCFG362P5E2V3I26AA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.448110838 (381h55m53.319193893s) [noise=true , address=145.79.227.37:4566 , piece_id=N6IMZ662JFVQWWVACNZYIGNCLFDKA3PF74XBRMQVIQFTJYU6FWHQ, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.448491503 (381h55m53.319574538s) [noise=true , address=77.90.31.10:4517 , piece_id=JKUFZKN4XVAW7BIVYPOQJURRNEG7I444RJSZC5IGH2X5FFQOJ2OA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.448573760 (381h55m53.319656805s) [noise=true , address=77.90.31.17:4773 , piece_id=ZYGPOGT7746PEFLPVTO5YO54ZDMA4I7XDYKOUAQPDHM3N6Z6E7OA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.448884412 (381h55m53.319967447s) [noise=true , address=77.90.31.14:4663 , piece_id=OLWF23WAL6KNLJ3SJKVJZC3BDXE2GREL3FPRNQATRK5DJEOICLEA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.448945598 (381h55m53.320028643s) [noise=true , address=145.79.227.48:4302 , piece_id=WEI5KLMAZX4TDRDHDYQP3MFMJD7NUPVYDVWKWNGQWHI72XQTNYBQ, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.449274494 (381h55m53.320357529s) [noise=true , address=145.79.227.38:4979 , piece_id=WBXCJPW3DXG7ICOO5QUG26LMS7G73NJAYANQLXOTPZ2WA4DRKHYA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.449325291 (381h55m53.320408336s) [noise=true , address=145.79.227.15:4874 , piece_id=A6FUVEMRY45O5DX34PGHNMLFPKESXU4E7RFIBZUHOOMZ6ZCKZF7Q, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.449651352 (381h55m53.320734387s) [noise=true , address=145.79.227.45:4230 , piece_id=G5UHDEEHX7WF6HXAWC2PYFVUQUGYRQKDRUAEMDNRVHJY4C7MDIJA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.449709993 (381h55m53.320793028s) [noise=true , address=145.79.227.32:4834 , piece_id=C25VCQMG32RWVXI3ROQPPURSYE7XTB3RHTCAVZ7N5DTTKK3JEY4Q, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.450191931 (381h55m53.321274966s) [noise=true , address=45.91.116.6:4974 , piece_id=MN5IF2YLUACGZTO4BG2UQWBCD3MLHDBGHYSHO5PUYVVLODSL7H5A, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.451236851 (381h55m53.322319886s) [noise=true , address=45.91.116.8:4046 , piece_id=L72GMUZ6FNO3ATO3OT2H2EZEUM3NATEGRKBSHUSGURSGETRV3BSA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:55.053012716 (381h55m54.924095751s) [upload=335186]: Next returned nil segment Dec 21 13:24:55.053027694 (381h55m54.924110729s) [upload=335186]: Waiting for error group managing segments... Dec 21 13:24:55.437142389 (381h55m55.308225424s) [noise=true , address=145.79.227.43:4157 , piece_id=CUGHWACL4LKBK3K5R6AYAU7PZSWTR5LH2KDP6JVV36XMFJVAQRAA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.438255078 (381h55m55.309338113s) [noise=true , address=145.79.227.45:4230 , piece_id=G5UHDEEHX7WF6HXAWC2PYFVUQUGYRQKDRUAEMDNRVHJY4C7MDIJA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.439457849 (381h55m55.310540884s) [noise=true , address=145.79.227.37:4566 , piece_id=N6IMZ662JFVQWWVACNZYIGNCLFDKA3PF74XBRMQVIQFTJYU6FWHQ, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.440351931 (381h55m55.311434967s) [noise=true , address=145.79.227.32:4834 , piece_id=C25VCQMG32RWVXI3ROQPPURSYE7XTB3RHTCAVZ7N5DTTKK3JEY4Q, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.441163436 (381h55m55.312246481s) [noise=true , address=145.79.227.48:4302 , piece_id=WEI5KLMAZX4TDRDHDYQP3MFMJD7NUPVYDVWKWNGQWHI72XQTNYBQ, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.458667117 (381h55m55.329750162s) [noise=true , address=145.79.227.15:4874 , piece_id=A6FUVEMRY45O5DX34PGHNMLFPKESXU4E7RFIBZUHOOMZ6ZCKZF7Q, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.459123205 (381h55m55.33020624s) [noise=true , address=145.79.227.51:4409 , piece_id=F2B7K7BE7JLBZMIHP5STKVAN7MH2PCYKJ33ZYJF4XJMABZSI75YA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.461487727 (381h55m55.332570773s) [noise=true , address=145.79.227.35:4481 , piece_id=BBT67CINYVQGAHO5U6KGRG6VDPVKUFQXMWWCFG362P5E2V3I26AA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.465407611 (381h55m55.336490656s) [noise=true , address=145.79.227.38:4979 , piece_id=WBXCJPW3DXG7ICOO5QUG26LMS7G73NJAYANQLXOTPZ2WA4DRKHYA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.691010826 (381h55m55.562093861s) [noise=true , address=77.90.31.14:4663 , piece_id=OLWF23WAL6KNLJ3SJKVJZC3BDXE2GREL3FPRNQATRK5DJEOICLEA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.715824785 (381h55m55.58690783s) [noise=true , address=45.91.116.6:4974 , piece_id=MN5IF2YLUACGZTO4BG2UQWBCD3MLHDBGHYSHO5PUYVVLODSL7H5A, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.734305105 (381h55m55.60538815s) [noise=true , address=45.91.116.8:4046 , piece_id=L72GMUZ6FNO3ATO3OT2H2EZEUM3NATEGRKBSHUSGURSGETRV3BSA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.917719464 (381h55m55.788802499s) [noise=true , address=77.90.31.10:4517 , piece_id=JKUFZKN4XVAW7BIVYPOQJURRNEG7I444RJSZC5IGH2X5FFQOJ2OA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.917738280 (381h55m55.788821315s) [seg_pos={0 0}, upload=335186]: Segment reached optimal threshold of 13 pieces. Dec 21 13:24:55.917957757 (381h55m55.789040802s) [noise=true , address=77.90.31.17:4773 , piece_id=ZYGPOGT7746PEFLPVTO5YO54ZDMA4I7XDYKOUAQPDHM3N6Z6E7OA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:50438->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:50438->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50438->77.90.31.17:4773: use of closed network connection Dec 21 13:24:55.917964471 (381h55m55.789047506s) [noise=true , address=77.90.31.17:4773 , piece_id=ZYGPOGT7746PEFLPVTO5YO54ZDMA4I7XDYKOUAQPDHM3N6Z6E7OA, seg_pos={0 0}, upload=335186]: Long tail cancellation due to optimal threshold Dec 21 13:24:55.918111541 (381h55m55.789194586s) [noise=true , address=145.79.227.44:4195 , piece_id=QECQLX7KZVU7RMFVZ6TOHRIXR776DFZ4D44FKFNUWYH2EURRYX7Q, seg_pos={0 0}, upload=335186]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 21 13:24:55.918121590 (381h55m55.789204635s) [noise=true , address=145.79.227.44:4195 , piece_id=QECQLX7KZVU7RMFVZ6TOHRIXR776DFZ4D44FKFNUWYH2EURRYX7Q, seg_pos={0 0}, upload=335186]: Long tail cancellation due to optimal threshold Dec 21 13:24:55.918156175 (381h55m55.789239221s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4CVCVL6PSTVUM33DZPLIU6QUN6JIXFCPBU6CNNA4P6YCHJ3PONA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:37888->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37888->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37888->45.91.116.13:4153: use of closed network connection Dec 21 13:24:55.918163128 (381h55m55.789246164s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4CVCVL6PSTVUM33DZPLIU6QUN6JIXFCPBU6CNNA4P6YCHJ3PONA, seg_pos={0 0}, upload=335186]: Long tail cancellation due to optimal threshold Dec 21 13:24:55.918263059 (381h55m55.789346094s) [seg_pos={0 0}, upload=335186]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 13:24:55.918332972 (381h55m55.789416017s) [upload=335186]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 13:24:55.918339284 (381h55m55.789422339s) [upload=335186]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 13:24:56.213644022 (381h55m56.084727067s) [upload=335186]: Done uploading segments... Dec 21 13:24:56.213663228 (381h55m56.084746263s) [upload=335186]: Upload finished. err: <nil> Duration: 6.150995535s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.X........X.....................XXXX...X..X...........X.....] duration: 5.896012343s trace: https://jaeger.dev.storj.io/trace/fa7962f52d9630d FAIL (2025-12-21 13:29:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: MN5IF2YLUACGZTO4BG2UQWBCD3MLHDBGHYSHO5PUYVVLODSL7H5A) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: JKUFZKN4XVAW7BIVYPOQJURRNEG7I444RJSZC5IGH2X5FFQOJ2OA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: L72GMUZ6FNO3ATO3OT2H2EZEUM3NATEGRKBSHUSGURSGETRV3BSA) context deadline exceeded ======LOGS====== gments... Dec 21 13:24:53.199397540 (381h55m53.070480575s) [upload=335186]: Got next segment. Inline: false Dec 21 13:24:53.199433087 (381h55m53.070516122s) [upload=335186]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 13:24:53.199437856 (381h55m53.070520901s) [upload=335186]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 13:24:53.446838205 (381h55m53.31792125s) [seg_pos={0 0}, upload=335186]: Begin upload segment... Dec 21 13:24:53.446894843 (381h55m53.317977878s) [seg_pos={0 0}, upload=335186]: Done begin upload segment. Dec 21 13:24:53.446931453 (381h55m53.318014488s) [noise=true , address=145.79.227.51:4409 , piece_id=F2B7K7BE7JLBZMIHP5STKVAN7MH2PCYKJ33ZYJF4XJMABZSI75YA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.446966390 (381h55m53.318049435s) [noise=true , address=145.79.227.43:4157 , piece_id=CUGHWACL4LKBK3K5R6AYAU7PZSWTR5LH2KDP6JVV36XMFJVAQRAA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.447612399 (381h55m53.318695444s) [noise=true , address=145.79.227.44:4195 , piece_id=QECQLX7KZVU7RMFVZ6TOHRIXR776DFZ4D44FKFNUWYH2EURRYX7Q, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.447623340 (381h55m53.318706395s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4CVCVL6PSTVUM33DZPLIU6QUN6JIXFCPBU6CNNA4P6YCHJ3PONA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.448102392 (381h55m53.319185437s) [noise=true , address=145.79.227.35:4481 , piece_id=BBT67CINYVQGAHO5U6KGRG6VDPVKUFQXMWWCFG362P5E2V3I26AA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.448110838 (381h55m53.319193893s) [noise=true , address=145.79.227.37:4566 , piece_id=N6IMZ662JFVQWWVACNZYIGNCLFDKA3PF74XBRMQVIQFTJYU6FWHQ, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.448491503 (381h55m53.319574538s) [noise=true , address=77.90.31.10:4517 , piece_id=JKUFZKN4XVAW7BIVYPOQJURRNEG7I444RJSZC5IGH2X5FFQOJ2OA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.448573760 (381h55m53.319656805s) [noise=true , address=77.90.31.17:4773 , piece_id=ZYGPOGT7746PEFLPVTO5YO54ZDMA4I7XDYKOUAQPDHM3N6Z6E7OA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.448884412 (381h55m53.319967447s) [noise=true , address=77.90.31.14:4663 , piece_id=OLWF23WAL6KNLJ3SJKVJZC3BDXE2GREL3FPRNQATRK5DJEOICLEA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.448945598 (381h55m53.320028643s) [noise=true , address=145.79.227.48:4302 , piece_id=WEI5KLMAZX4TDRDHDYQP3MFMJD7NUPVYDVWKWNGQWHI72XQTNYBQ, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.449274494 (381h55m53.320357529s) [noise=true , address=145.79.227.38:4979 , piece_id=WBXCJPW3DXG7ICOO5QUG26LMS7G73NJAYANQLXOTPZ2WA4DRKHYA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.449325291 (381h55m53.320408336s) [noise=true , address=145.79.227.15:4874 , piece_id=A6FUVEMRY45O5DX34PGHNMLFPKESXU4E7RFIBZUHOOMZ6ZCKZF7Q, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.449651352 (381h55m53.320734387s) [noise=true , address=145.79.227.45:4230 , piece_id=G5UHDEEHX7WF6HXAWC2PYFVUQUGYRQKDRUAEMDNRVHJY4C7MDIJA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.449709993 (381h55m53.320793028s) [noise=true , address=145.79.227.32:4834 , piece_id=C25VCQMG32RWVXI3ROQPPURSYE7XTB3RHTCAVZ7N5DTTKK3JEY4Q, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.450191931 (381h55m53.321274966s) [noise=true , address=45.91.116.6:4974 , piece_id=MN5IF2YLUACGZTO4BG2UQWBCD3MLHDBGHYSHO5PUYVVLODSL7H5A, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.451236851 (381h55m53.322319886s) [noise=true , address=45.91.116.8:4046 , piece_id=L72GMUZ6FNO3ATO3OT2H2EZEUM3NATEGRKBSHUSGURSGETRV3BSA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:55.053012716 (381h55m54.924095751s) [upload=335186]: Next returned nil segment Dec 21 13:24:55.053027694 (381h55m54.924110729s) [upload=335186]: Waiting for error group managing segments... Dec 21 13:24:55.437142389 (381h55m55.308225424s) [noise=true , address=145.79.227.43:4157 , piece_id=CUGHWACL4LKBK3K5R6AYAU7PZSWTR5LH2KDP6JVV36XMFJVAQRAA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.438255078 (381h55m55.309338113s) [noise=true , address=145.79.227.45:4230 , piece_id=G5UHDEEHX7WF6HXAWC2PYFVUQUGYRQKDRUAEMDNRVHJY4C7MDIJA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.439457849 (381h55m55.310540884s) [noise=true , address=145.79.227.37:4566 , piece_id=N6IMZ662JFVQWWVACNZYIGNCLFDKA3PF74XBRMQVIQFTJYU6FWHQ, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.440351931 (381h55m55.311434967s) [noise=true , address=145.79.227.32:4834 , piece_id=C25VCQMG32RWVXI3ROQPPURSYE7XTB3RHTCAVZ7N5DTTKK3JEY4Q, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.441163436 (381h55m55.312246481s) [noise=true , address=145.79.227.48:4302 , piece_id=WEI5KLMAZX4TDRDHDYQP3MFMJD7NUPVYDVWKWNGQWHI72XQTNYBQ, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.458667117 (381h55m55.329750162s) [noise=true , address=145.79.227.15:4874 , piece_id=A6FUVEMRY45O5DX34PGHNMLFPKESXU4E7RFIBZUHOOMZ6ZCKZF7Q, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.459123205 (381h55m55.33020624s) [noise=true , address=145.79.227.51:4409 , piece_id=F2B7K7BE7JLBZMIHP5STKVAN7MH2PCYKJ33ZYJF4XJMABZSI75YA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.461487727 (381h55m55.332570773s) [noise=true , address=145.79.227.35:4481 , piece_id=BBT67CINYVQGAHO5U6KGRG6VDPVKUFQXMWWCFG362P5E2V3I26AA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.465407611 (381h55m55.336490656s) [noise=true , address=145.79.227.38:4979 , piece_id=WBXCJPW3DXG7ICOO5QUG26LMS7G73NJAYANQLXOTPZ2WA4DRKHYA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.691010826 (381h55m55.562093861s) [noise=true , address=77.90.31.14:4663 , piece_id=OLWF23WAL6KNLJ3SJKVJZC3BDXE2GREL3FPRNQATRK5DJEOICLEA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.715824785 (381h55m55.58690783s) [noise=true , address=45.91.116.6:4974 , piece_id=MN5IF2YLUACGZTO4BG2UQWBCD3MLHDBGHYSHO5PUYVVLODSL7H5A, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.734305105 (381h55m55.60538815s) [noise=true , address=45.91.116.8:4046 , piece_id=L72GMUZ6FNO3ATO3OT2H2EZEUM3NATEGRKBSHUSGURSGETRV3BSA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.917719464 (381h55m55.788802499s) [noise=true , address=77.90.31.10:4517 , piece_id=JKUFZKN4XVAW7BIVYPOQJURRNEG7I444RJSZC5IGH2X5FFQOJ2OA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.917738280 (381h55m55.788821315s) [seg_pos={0 0}, upload=335186]: Segment reached optimal threshold of 13 pieces. Dec 21 13:24:55.917957757 (381h55m55.789040802s) [noise=true , address=77.90.31.17:4773 , piece_id=ZYGPOGT7746PEFLPVTO5YO54ZDMA4I7XDYKOUAQPDHM3N6Z6E7OA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:50438->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:50438->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50438->77.90.31.17:4773: use of closed network connection Dec 21 13:24:55.917964471 (381h55m55.789047506s) [noise=true , address=77.90.31.17:4773 , piece_id=ZYGPOGT7746PEFLPVTO5YO54ZDMA4I7XDYKOUAQPDHM3N6Z6E7OA, seg_pos={0 0}, upload=335186]: Long tail cancellation due to optimal threshold Dec 21 13:24:55.918111541 (381h55m55.789194586s) [noise=true , address=145.79.227.44:4195 , piece_id=QECQLX7KZVU7RMFVZ6TOHRIXR776DFZ4D44FKFNUWYH2EURRYX7Q, seg_pos={0 0}, upload=335186]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 21 13:24:55.918121590 (381h55m55.789204635s) [noise=true , address=145.79.227.44:4195 , piece_id=QECQLX7KZVU7RMFVZ6TOHRIXR776DFZ4D44FKFNUWYH2EURRYX7Q, seg_pos={0 0}, upload=335186]: Long tail cancellation due to optimal threshold Dec 21 13:24:55.918156175 (381h55m55.789239221s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4CVCVL6PSTVUM33DZPLIU6QUN6JIXFCPBU6CNNA4P6YCHJ3PONA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:37888->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37888->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37888->45.91.116.13:4153: use of closed network connection Dec 21 13:24:55.918163128 (381h55m55.789246164s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4CVCVL6PSTVUM33DZPLIU6QUN6JIXFCPBU6CNNA4P6YCHJ3PONA, seg_pos={0 0}, upload=335186]: Long tail cancellation due to optimal threshold Dec 21 13:24:55.918263059 (381h55m55.789346094s) [seg_pos={0 0}, upload=335186]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 13:24:55.918332972 (381h55m55.789416017s) [upload=335186]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 13:24:55.918339284 (381h55m55.789422339s) [upload=335186]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 13:24:56.213644022 (381h55m56.084727067s) [upload=335186]: Done uploading segments... Dec 21 13:24:56.213663228 (381h55m56.084746263s) [upload=335186]: Upload finished. err: <nil> Duration: 5.896042872s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..X........X.....................XXXX...X..X...........X....] duration: 6.018937622s trace: https://jaeger.dev.storj.io/trace/fa7962f52d9630d FAIL (2025-12-21 13:27:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: MN5IF2YLUACGZTO4BG2UQWBCD3MLHDBGHYSHO5PUYVVLODSL7H5A) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: JKUFZKN4XVAW7BIVYPOQJURRNEG7I444RJSZC5IGH2X5FFQOJ2OA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: L72GMUZ6FNO3ATO3OT2H2EZEUM3NATEGRKBSHUSGURSGETRV3BSA) context deadline exceeded ======LOGS====== gments... Dec 21 13:24:53.199397540 (381h55m53.070480575s) [upload=335186]: Got next segment. Inline: false Dec 21 13:24:53.199433087 (381h55m53.070516122s) [upload=335186]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 13:24:53.199437856 (381h55m53.070520901s) [upload=335186]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 13:24:53.446838205 (381h55m53.31792125s) [seg_pos={0 0}, upload=335186]: Begin upload segment... Dec 21 13:24:53.446894843 (381h55m53.317977878s) [seg_pos={0 0}, upload=335186]: Done begin upload segment. Dec 21 13:24:53.446931453 (381h55m53.318014488s) [noise=true , address=145.79.227.51:4409 , piece_id=F2B7K7BE7JLBZMIHP5STKVAN7MH2PCYKJ33ZYJF4XJMABZSI75YA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.446966390 (381h55m53.318049435s) [noise=true , address=145.79.227.43:4157 , piece_id=CUGHWACL4LKBK3K5R6AYAU7PZSWTR5LH2KDP6JVV36XMFJVAQRAA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.447612399 (381h55m53.318695444s) [noise=true , address=145.79.227.44:4195 , piece_id=QECQLX7KZVU7RMFVZ6TOHRIXR776DFZ4D44FKFNUWYH2EURRYX7Q, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.447623340 (381h55m53.318706395s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4CVCVL6PSTVUM33DZPLIU6QUN6JIXFCPBU6CNNA4P6YCHJ3PONA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.448102392 (381h55m53.319185437s) [noise=true , address=145.79.227.35:4481 , piece_id=BBT67CINYVQGAHO5U6KGRG6VDPVKUFQXMWWCFG362P5E2V3I26AA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.448110838 (381h55m53.319193893s) [noise=true , address=145.79.227.37:4566 , piece_id=N6IMZ662JFVQWWVACNZYIGNCLFDKA3PF74XBRMQVIQFTJYU6FWHQ, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.448491503 (381h55m53.319574538s) [noise=true , address=77.90.31.10:4517 , piece_id=JKUFZKN4XVAW7BIVYPOQJURRNEG7I444RJSZC5IGH2X5FFQOJ2OA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.448573760 (381h55m53.319656805s) [noise=true , address=77.90.31.17:4773 , piece_id=ZYGPOGT7746PEFLPVTO5YO54ZDMA4I7XDYKOUAQPDHM3N6Z6E7OA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.448884412 (381h55m53.319967447s) [noise=true , address=77.90.31.14:4663 , piece_id=OLWF23WAL6KNLJ3SJKVJZC3BDXE2GREL3FPRNQATRK5DJEOICLEA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.448945598 (381h55m53.320028643s) [noise=true , address=145.79.227.48:4302 , piece_id=WEI5KLMAZX4TDRDHDYQP3MFMJD7NUPVYDVWKWNGQWHI72XQTNYBQ, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.449274494 (381h55m53.320357529s) [noise=true , address=145.79.227.38:4979 , piece_id=WBXCJPW3DXG7ICOO5QUG26LMS7G73NJAYANQLXOTPZ2WA4DRKHYA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.449325291 (381h55m53.320408336s) [noise=true , address=145.79.227.15:4874 , piece_id=A6FUVEMRY45O5DX34PGHNMLFPKESXU4E7RFIBZUHOOMZ6ZCKZF7Q, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.449651352 (381h55m53.320734387s) [noise=true , address=145.79.227.45:4230 , piece_id=G5UHDEEHX7WF6HXAWC2PYFVUQUGYRQKDRUAEMDNRVHJY4C7MDIJA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.449709993 (381h55m53.320793028s) [noise=true , address=145.79.227.32:4834 , piece_id=C25VCQMG32RWVXI3ROQPPURSYE7XTB3RHTCAVZ7N5DTTKK3JEY4Q, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.450191931 (381h55m53.321274966s) [noise=true , address=45.91.116.6:4974 , piece_id=MN5IF2YLUACGZTO4BG2UQWBCD3MLHDBGHYSHO5PUYVVLODSL7H5A, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.451236851 (381h55m53.322319886s) [noise=true , address=45.91.116.8:4046 , piece_id=L72GMUZ6FNO3ATO3OT2H2EZEUM3NATEGRKBSHUSGURSGETRV3BSA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:55.053012716 (381h55m54.924095751s) [upload=335186]: Next returned nil segment Dec 21 13:24:55.053027694 (381h55m54.924110729s) [upload=335186]: Waiting for error group managing segments... Dec 21 13:24:55.437142389 (381h55m55.308225424s) [noise=true , address=145.79.227.43:4157 , piece_id=CUGHWACL4LKBK3K5R6AYAU7PZSWTR5LH2KDP6JVV36XMFJVAQRAA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.438255078 (381h55m55.309338113s) [noise=true , address=145.79.227.45:4230 , piece_id=G5UHDEEHX7WF6HXAWC2PYFVUQUGYRQKDRUAEMDNRVHJY4C7MDIJA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.439457849 (381h55m55.310540884s) [noise=true , address=145.79.227.37:4566 , piece_id=N6IMZ662JFVQWWVACNZYIGNCLFDKA3PF74XBRMQVIQFTJYU6FWHQ, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.440351931 (381h55m55.311434967s) [noise=true , address=145.79.227.32:4834 , piece_id=C25VCQMG32RWVXI3ROQPPURSYE7XTB3RHTCAVZ7N5DTTKK3JEY4Q, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.441163436 (381h55m55.312246481s) [noise=true , address=145.79.227.48:4302 , piece_id=WEI5KLMAZX4TDRDHDYQP3MFMJD7NUPVYDVWKWNGQWHI72XQTNYBQ, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.458667117 (381h55m55.329750162s) [noise=true , address=145.79.227.15:4874 , piece_id=A6FUVEMRY45O5DX34PGHNMLFPKESXU4E7RFIBZUHOOMZ6ZCKZF7Q, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.459123205 (381h55m55.33020624s) [noise=true , address=145.79.227.51:4409 , piece_id=F2B7K7BE7JLBZMIHP5STKVAN7MH2PCYKJ33ZYJF4XJMABZSI75YA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.461487727 (381h55m55.332570773s) [noise=true , address=145.79.227.35:4481 , piece_id=BBT67CINYVQGAHO5U6KGRG6VDPVKUFQXMWWCFG362P5E2V3I26AA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.465407611 (381h55m55.336490656s) [noise=true , address=145.79.227.38:4979 , piece_id=WBXCJPW3DXG7ICOO5QUG26LMS7G73NJAYANQLXOTPZ2WA4DRKHYA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.691010826 (381h55m55.562093861s) [noise=true , address=77.90.31.14:4663 , piece_id=OLWF23WAL6KNLJ3SJKVJZC3BDXE2GREL3FPRNQATRK5DJEOICLEA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.715824785 (381h55m55.58690783s) [noise=true , address=45.91.116.6:4974 , piece_id=MN5IF2YLUACGZTO4BG2UQWBCD3MLHDBGHYSHO5PUYVVLODSL7H5A, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.734305105 (381h55m55.60538815s) [noise=true , address=45.91.116.8:4046 , piece_id=L72GMUZ6FNO3ATO3OT2H2EZEUM3NATEGRKBSHUSGURSGETRV3BSA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.917719464 (381h55m55.788802499s) [noise=true , address=77.90.31.10:4517 , piece_id=JKUFZKN4XVAW7BIVYPOQJURRNEG7I444RJSZC5IGH2X5FFQOJ2OA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.917738280 (381h55m55.788821315s) [seg_pos={0 0}, upload=335186]: Segment reached optimal threshold of 13 pieces. Dec 21 13:24:55.917957757 (381h55m55.789040802s) [noise=true , address=77.90.31.17:4773 , piece_id=ZYGPOGT7746PEFLPVTO5YO54ZDMA4I7XDYKOUAQPDHM3N6Z6E7OA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:50438->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:50438->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50438->77.90.31.17:4773: use of closed network connection Dec 21 13:24:55.917964471 (381h55m55.789047506s) [noise=true , address=77.90.31.17:4773 , piece_id=ZYGPOGT7746PEFLPVTO5YO54ZDMA4I7XDYKOUAQPDHM3N6Z6E7OA, seg_pos={0 0}, upload=335186]: Long tail cancellation due to optimal threshold Dec 21 13:24:55.918111541 (381h55m55.789194586s) [noise=true , address=145.79.227.44:4195 , piece_id=QECQLX7KZVU7RMFVZ6TOHRIXR776DFZ4D44FKFNUWYH2EURRYX7Q, seg_pos={0 0}, upload=335186]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 21 13:24:55.918121590 (381h55m55.789204635s) [noise=true , address=145.79.227.44:4195 , piece_id=QECQLX7KZVU7RMFVZ6TOHRIXR776DFZ4D44FKFNUWYH2EURRYX7Q, seg_pos={0 0}, upload=335186]: Long tail cancellation due to optimal threshold Dec 21 13:24:55.918156175 (381h55m55.789239221s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4CVCVL6PSTVUM33DZPLIU6QUN6JIXFCPBU6CNNA4P6YCHJ3PONA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:37888->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37888->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37888->45.91.116.13:4153: use of closed network connection Dec 21 13:24:55.918163128 (381h55m55.789246164s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4CVCVL6PSTVUM33DZPLIU6QUN6JIXFCPBU6CNNA4P6YCHJ3PONA, seg_pos={0 0}, upload=335186]: Long tail cancellation due to optimal threshold Dec 21 13:24:55.918263059 (381h55m55.789346094s) [seg_pos={0 0}, upload=335186]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 13:24:55.918332972 (381h55m55.789416017s) [upload=335186]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 13:24:55.918339284 (381h55m55.789422339s) [upload=335186]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 13:24:56.213644022 (381h55m56.084727067s) [upload=335186]: Done uploading segments... Dec 21 13:24:56.213663228 (381h55m56.084746263s) [upload=335186]: Upload finished. err: <nil> Duration: 6.018963142s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...X........X.....................XXXX...X..X...........X...] duration: 6.118536034s trace: https://jaeger.dev.storj.io/trace/fa7962f52d9630d FAIL (2025-12-21 13:27:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: MN5IF2YLUACGZTO4BG2UQWBCD3MLHDBGHYSHO5PUYVVLODSL7H5A) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: JKUFZKN4XVAW7BIVYPOQJURRNEG7I444RJSZC5IGH2X5FFQOJ2OA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: L72GMUZ6FNO3ATO3OT2H2EZEUM3NATEGRKBSHUSGURSGETRV3BSA) context deadline exceeded ======LOGS====== gments... Dec 21 13:24:53.199397540 (381h55m53.070480575s) [upload=335186]: Got next segment. Inline: false Dec 21 13:24:53.199433087 (381h55m53.070516122s) [upload=335186]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 13:24:53.199437856 (381h55m53.070520901s) [upload=335186]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 13:24:53.446838205 (381h55m53.31792125s) [seg_pos={0 0}, upload=335186]: Begin upload segment... Dec 21 13:24:53.446894843 (381h55m53.317977878s) [seg_pos={0 0}, upload=335186]: Done begin upload segment. Dec 21 13:24:53.446931453 (381h55m53.318014488s) [noise=true , address=145.79.227.51:4409 , piece_id=F2B7K7BE7JLBZMIHP5STKVAN7MH2PCYKJ33ZYJF4XJMABZSI75YA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.446966390 (381h55m53.318049435s) [noise=true , address=145.79.227.43:4157 , piece_id=CUGHWACL4LKBK3K5R6AYAU7PZSWTR5LH2KDP6JVV36XMFJVAQRAA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.447612399 (381h55m53.318695444s) [noise=true , address=145.79.227.44:4195 , piece_id=QECQLX7KZVU7RMFVZ6TOHRIXR776DFZ4D44FKFNUWYH2EURRYX7Q, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.447623340 (381h55m53.318706395s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4CVCVL6PSTVUM33DZPLIU6QUN6JIXFCPBU6CNNA4P6YCHJ3PONA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.448102392 (381h55m53.319185437s) [noise=true , address=145.79.227.35:4481 , piece_id=BBT67CINYVQGAHO5U6KGRG6VDPVKUFQXMWWCFG362P5E2V3I26AA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.448110838 (381h55m53.319193893s) [noise=true , address=145.79.227.37:4566 , piece_id=N6IMZ662JFVQWWVACNZYIGNCLFDKA3PF74XBRMQVIQFTJYU6FWHQ, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.448491503 (381h55m53.319574538s) [noise=true , address=77.90.31.10:4517 , piece_id=JKUFZKN4XVAW7BIVYPOQJURRNEG7I444RJSZC5IGH2X5FFQOJ2OA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.448573760 (381h55m53.319656805s) [noise=true , address=77.90.31.17:4773 , piece_id=ZYGPOGT7746PEFLPVTO5YO54ZDMA4I7XDYKOUAQPDHM3N6Z6E7OA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.448884412 (381h55m53.319967447s) [noise=true , address=77.90.31.14:4663 , piece_id=OLWF23WAL6KNLJ3SJKVJZC3BDXE2GREL3FPRNQATRK5DJEOICLEA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.448945598 (381h55m53.320028643s) [noise=true , address=145.79.227.48:4302 , piece_id=WEI5KLMAZX4TDRDHDYQP3MFMJD7NUPVYDVWKWNGQWHI72XQTNYBQ, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.449274494 (381h55m53.320357529s) [noise=true , address=145.79.227.38:4979 , piece_id=WBXCJPW3DXG7ICOO5QUG26LMS7G73NJAYANQLXOTPZ2WA4DRKHYA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.449325291 (381h55m53.320408336s) [noise=true , address=145.79.227.15:4874 , piece_id=A6FUVEMRY45O5DX34PGHNMLFPKESXU4E7RFIBZUHOOMZ6ZCKZF7Q, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.449651352 (381h55m53.320734387s) [noise=true , address=145.79.227.45:4230 , piece_id=G5UHDEEHX7WF6HXAWC2PYFVUQUGYRQKDRUAEMDNRVHJY4C7MDIJA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.449709993 (381h55m53.320793028s) [noise=true , address=145.79.227.32:4834 , piece_id=C25VCQMG32RWVXI3ROQPPURSYE7XTB3RHTCAVZ7N5DTTKK3JEY4Q, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.450191931 (381h55m53.321274966s) [noise=true , address=45.91.116.6:4974 , piece_id=MN5IF2YLUACGZTO4BG2UQWBCD3MLHDBGHYSHO5PUYVVLODSL7H5A, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.451236851 (381h55m53.322319886s) [noise=true , address=45.91.116.8:4046 , piece_id=L72GMUZ6FNO3ATO3OT2H2EZEUM3NATEGRKBSHUSGURSGETRV3BSA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:55.053012716 (381h55m54.924095751s) [upload=335186]: Next returned nil segment Dec 21 13:24:55.053027694 (381h55m54.924110729s) [upload=335186]: Waiting for error group managing segments... Dec 21 13:24:55.437142389 (381h55m55.308225424s) [noise=true , address=145.79.227.43:4157 , piece_id=CUGHWACL4LKBK3K5R6AYAU7PZSWTR5LH2KDP6JVV36XMFJVAQRAA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.438255078 (381h55m55.309338113s) [noise=true , address=145.79.227.45:4230 , piece_id=G5UHDEEHX7WF6HXAWC2PYFVUQUGYRQKDRUAEMDNRVHJY4C7MDIJA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.439457849 (381h55m55.310540884s) [noise=true , address=145.79.227.37:4566 , piece_id=N6IMZ662JFVQWWVACNZYIGNCLFDKA3PF74XBRMQVIQFTJYU6FWHQ, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.440351931 (381h55m55.311434967s) [noise=true , address=145.79.227.32:4834 , piece_id=C25VCQMG32RWVXI3ROQPPURSYE7XTB3RHTCAVZ7N5DTTKK3JEY4Q, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.441163436 (381h55m55.312246481s) [noise=true , address=145.79.227.48:4302 , piece_id=WEI5KLMAZX4TDRDHDYQP3MFMJD7NUPVYDVWKWNGQWHI72XQTNYBQ, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.458667117 (381h55m55.329750162s) [noise=true , address=145.79.227.15:4874 , piece_id=A6FUVEMRY45O5DX34PGHNMLFPKESXU4E7RFIBZUHOOMZ6ZCKZF7Q, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.459123205 (381h55m55.33020624s) [noise=true , address=145.79.227.51:4409 , piece_id=F2B7K7BE7JLBZMIHP5STKVAN7MH2PCYKJ33ZYJF4XJMABZSI75YA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.461487727 (381h55m55.332570773s) [noise=true , address=145.79.227.35:4481 , piece_id=BBT67CINYVQGAHO5U6KGRG6VDPVKUFQXMWWCFG362P5E2V3I26AA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.465407611 (381h55m55.336490656s) [noise=true , address=145.79.227.38:4979 , piece_id=WBXCJPW3DXG7ICOO5QUG26LMS7G73NJAYANQLXOTPZ2WA4DRKHYA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.691010826 (381h55m55.562093861s) [noise=true , address=77.90.31.14:4663 , piece_id=OLWF23WAL6KNLJ3SJKVJZC3BDXE2GREL3FPRNQATRK5DJEOICLEA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.715824785 (381h55m55.58690783s) [noise=true , address=45.91.116.6:4974 , piece_id=MN5IF2YLUACGZTO4BG2UQWBCD3MLHDBGHYSHO5PUYVVLODSL7H5A, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.734305105 (381h55m55.60538815s) [noise=true , address=45.91.116.8:4046 , piece_id=L72GMUZ6FNO3ATO3OT2H2EZEUM3NATEGRKBSHUSGURSGETRV3BSA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.917719464 (381h55m55.788802499s) [noise=true , address=77.90.31.10:4517 , piece_id=JKUFZKN4XVAW7BIVYPOQJURRNEG7I444RJSZC5IGH2X5FFQOJ2OA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.917738280 (381h55m55.788821315s) [seg_pos={0 0}, upload=335186]: Segment reached optimal threshold of 13 pieces. Dec 21 13:24:55.917957757 (381h55m55.789040802s) [noise=true , address=77.90.31.17:4773 , piece_id=ZYGPOGT7746PEFLPVTO5YO54ZDMA4I7XDYKOUAQPDHM3N6Z6E7OA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:50438->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:50438->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50438->77.90.31.17:4773: use of closed network connection Dec 21 13:24:55.917964471 (381h55m55.789047506s) [noise=true , address=77.90.31.17:4773 , piece_id=ZYGPOGT7746PEFLPVTO5YO54ZDMA4I7XDYKOUAQPDHM3N6Z6E7OA, seg_pos={0 0}, upload=335186]: Long tail cancellation due to optimal threshold Dec 21 13:24:55.918111541 (381h55m55.789194586s) [noise=true , address=145.79.227.44:4195 , piece_id=QECQLX7KZVU7RMFVZ6TOHRIXR776DFZ4D44FKFNUWYH2EURRYX7Q, seg_pos={0 0}, upload=335186]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 21 13:24:55.918121590 (381h55m55.789204635s) [noise=true , address=145.79.227.44:4195 , piece_id=QECQLX7KZVU7RMFVZ6TOHRIXR776DFZ4D44FKFNUWYH2EURRYX7Q, seg_pos={0 0}, upload=335186]: Long tail cancellation due to optimal threshold Dec 21 13:24:55.918156175 (381h55m55.789239221s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4CVCVL6PSTVUM33DZPLIU6QUN6JIXFCPBU6CNNA4P6YCHJ3PONA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:37888->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37888->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37888->45.91.116.13:4153: use of closed network connection Dec 21 13:24:55.918163128 (381h55m55.789246164s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4CVCVL6PSTVUM33DZPLIU6QUN6JIXFCPBU6CNNA4P6YCHJ3PONA, seg_pos={0 0}, upload=335186]: Long tail cancellation due to optimal threshold Dec 21 13:24:55.918263059 (381h55m55.789346094s) [seg_pos={0 0}, upload=335186]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 13:24:55.918332972 (381h55m55.789416017s) [upload=335186]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 13:24:55.918339284 (381h55m55.789422339s) [upload=335186]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 13:24:56.213644022 (381h55m56.084727067s) [upload=335186]: Done uploading segments... Dec 21 13:24:56.213663228 (381h55m56.084746263s) [upload=335186]: Upload finished. err: <nil> Duration: 6.118563826s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [....X........X.....................XXXX...X..X...........X..] duration: 7.505751666s trace: https://jaeger.dev.storj.io/trace/fa7962f52d9630d FAIL (2025-12-21 13:26:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: MN5IF2YLUACGZTO4BG2UQWBCD3MLHDBGHYSHO5PUYVVLODSL7H5A) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: JKUFZKN4XVAW7BIVYPOQJURRNEG7I444RJSZC5IGH2X5FFQOJ2OA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: L72GMUZ6FNO3ATO3OT2H2EZEUM3NATEGRKBSHUSGURSGETRV3BSA) context deadline exceeded ======LOGS====== gments... Dec 21 13:24:53.199397540 (381h55m53.070480575s) [upload=335186]: Got next segment. Inline: false Dec 21 13:24:53.199433087 (381h55m53.070516122s) [upload=335186]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 13:24:53.199437856 (381h55m53.070520901s) [upload=335186]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 13:24:53.446838205 (381h55m53.31792125s) [seg_pos={0 0}, upload=335186]: Begin upload segment... Dec 21 13:24:53.446894843 (381h55m53.317977878s) [seg_pos={0 0}, upload=335186]: Done begin upload segment. Dec 21 13:24:53.446931453 (381h55m53.318014488s) [noise=true , address=145.79.227.51:4409 , piece_id=F2B7K7BE7JLBZMIHP5STKVAN7MH2PCYKJ33ZYJF4XJMABZSI75YA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.446966390 (381h55m53.318049435s) [noise=true , address=145.79.227.43:4157 , piece_id=CUGHWACL4LKBK3K5R6AYAU7PZSWTR5LH2KDP6JVV36XMFJVAQRAA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.447612399 (381h55m53.318695444s) [noise=true , address=145.79.227.44:4195 , piece_id=QECQLX7KZVU7RMFVZ6TOHRIXR776DFZ4D44FKFNUWYH2EURRYX7Q, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.447623340 (381h55m53.318706395s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4CVCVL6PSTVUM33DZPLIU6QUN6JIXFCPBU6CNNA4P6YCHJ3PONA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.448102392 (381h55m53.319185437s) [noise=true , address=145.79.227.35:4481 , piece_id=BBT67CINYVQGAHO5U6KGRG6VDPVKUFQXMWWCFG362P5E2V3I26AA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.448110838 (381h55m53.319193893s) [noise=true , address=145.79.227.37:4566 , piece_id=N6IMZ662JFVQWWVACNZYIGNCLFDKA3PF74XBRMQVIQFTJYU6FWHQ, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.448491503 (381h55m53.319574538s) [noise=true , address=77.90.31.10:4517 , piece_id=JKUFZKN4XVAW7BIVYPOQJURRNEG7I444RJSZC5IGH2X5FFQOJ2OA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.448573760 (381h55m53.319656805s) [noise=true , address=77.90.31.17:4773 , piece_id=ZYGPOGT7746PEFLPVTO5YO54ZDMA4I7XDYKOUAQPDHM3N6Z6E7OA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.448884412 (381h55m53.319967447s) [noise=true , address=77.90.31.14:4663 , piece_id=OLWF23WAL6KNLJ3SJKVJZC3BDXE2GREL3FPRNQATRK5DJEOICLEA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.448945598 (381h55m53.320028643s) [noise=true , address=145.79.227.48:4302 , piece_id=WEI5KLMAZX4TDRDHDYQP3MFMJD7NUPVYDVWKWNGQWHI72XQTNYBQ, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.449274494 (381h55m53.320357529s) [noise=true , address=145.79.227.38:4979 , piece_id=WBXCJPW3DXG7ICOO5QUG26LMS7G73NJAYANQLXOTPZ2WA4DRKHYA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.449325291 (381h55m53.320408336s) [noise=true , address=145.79.227.15:4874 , piece_id=A6FUVEMRY45O5DX34PGHNMLFPKESXU4E7RFIBZUHOOMZ6ZCKZF7Q, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.449651352 (381h55m53.320734387s) [noise=true , address=145.79.227.45:4230 , piece_id=G5UHDEEHX7WF6HXAWC2PYFVUQUGYRQKDRUAEMDNRVHJY4C7MDIJA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.449709993 (381h55m53.320793028s) [noise=true , address=145.79.227.32:4834 , piece_id=C25VCQMG32RWVXI3ROQPPURSYE7XTB3RHTCAVZ7N5DTTKK3JEY4Q, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.450191931 (381h55m53.321274966s) [noise=true , address=45.91.116.6:4974 , piece_id=MN5IF2YLUACGZTO4BG2UQWBCD3MLHDBGHYSHO5PUYVVLODSL7H5A, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.451236851 (381h55m53.322319886s) [noise=true , address=45.91.116.8:4046 , piece_id=L72GMUZ6FNO3ATO3OT2H2EZEUM3NATEGRKBSHUSGURSGETRV3BSA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:55.053012716 (381h55m54.924095751s) [upload=335186]: Next returned nil segment Dec 21 13:24:55.053027694 (381h55m54.924110729s) [upload=335186]: Waiting for error group managing segments... Dec 21 13:24:55.437142389 (381h55m55.308225424s) [noise=true , address=145.79.227.43:4157 , piece_id=CUGHWACL4LKBK3K5R6AYAU7PZSWTR5LH2KDP6JVV36XMFJVAQRAA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.438255078 (381h55m55.309338113s) [noise=true , address=145.79.227.45:4230 , piece_id=G5UHDEEHX7WF6HXAWC2PYFVUQUGYRQKDRUAEMDNRVHJY4C7MDIJA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.439457849 (381h55m55.310540884s) [noise=true , address=145.79.227.37:4566 , piece_id=N6IMZ662JFVQWWVACNZYIGNCLFDKA3PF74XBRMQVIQFTJYU6FWHQ, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.440351931 (381h55m55.311434967s) [noise=true , address=145.79.227.32:4834 , piece_id=C25VCQMG32RWVXI3ROQPPURSYE7XTB3RHTCAVZ7N5DTTKK3JEY4Q, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.441163436 (381h55m55.312246481s) [noise=true , address=145.79.227.48:4302 , piece_id=WEI5KLMAZX4TDRDHDYQP3MFMJD7NUPVYDVWKWNGQWHI72XQTNYBQ, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.458667117 (381h55m55.329750162s) [noise=true , address=145.79.227.15:4874 , piece_id=A6FUVEMRY45O5DX34PGHNMLFPKESXU4E7RFIBZUHOOMZ6ZCKZF7Q, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.459123205 (381h55m55.33020624s) [noise=true , address=145.79.227.51:4409 , piece_id=F2B7K7BE7JLBZMIHP5STKVAN7MH2PCYKJ33ZYJF4XJMABZSI75YA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.461487727 (381h55m55.332570773s) [noise=true , address=145.79.227.35:4481 , piece_id=BBT67CINYVQGAHO5U6KGRG6VDPVKUFQXMWWCFG362P5E2V3I26AA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.465407611 (381h55m55.336490656s) [noise=true , address=145.79.227.38:4979 , piece_id=WBXCJPW3DXG7ICOO5QUG26LMS7G73NJAYANQLXOTPZ2WA4DRKHYA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.691010826 (381h55m55.562093861s) [noise=true , address=77.90.31.14:4663 , piece_id=OLWF23WAL6KNLJ3SJKVJZC3BDXE2GREL3FPRNQATRK5DJEOICLEA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.715824785 (381h55m55.58690783s) [noise=true , address=45.91.116.6:4974 , piece_id=MN5IF2YLUACGZTO4BG2UQWBCD3MLHDBGHYSHO5PUYVVLODSL7H5A, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.734305105 (381h55m55.60538815s) [noise=true , address=45.91.116.8:4046 , piece_id=L72GMUZ6FNO3ATO3OT2H2EZEUM3NATEGRKBSHUSGURSGETRV3BSA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.917719464 (381h55m55.788802499s) [noise=true , address=77.90.31.10:4517 , piece_id=JKUFZKN4XVAW7BIVYPOQJURRNEG7I444RJSZC5IGH2X5FFQOJ2OA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.917738280 (381h55m55.788821315s) [seg_pos={0 0}, upload=335186]: Segment reached optimal threshold of 13 pieces. Dec 21 13:24:55.917957757 (381h55m55.789040802s) [noise=true , address=77.90.31.17:4773 , piece_id=ZYGPOGT7746PEFLPVTO5YO54ZDMA4I7XDYKOUAQPDHM3N6Z6E7OA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:50438->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:50438->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50438->77.90.31.17:4773: use of closed network connection Dec 21 13:24:55.917964471 (381h55m55.789047506s) [noise=true , address=77.90.31.17:4773 , piece_id=ZYGPOGT7746PEFLPVTO5YO54ZDMA4I7XDYKOUAQPDHM3N6Z6E7OA, seg_pos={0 0}, upload=335186]: Long tail cancellation due to optimal threshold Dec 21 13:24:55.918111541 (381h55m55.789194586s) [noise=true , address=145.79.227.44:4195 , piece_id=QECQLX7KZVU7RMFVZ6TOHRIXR776DFZ4D44FKFNUWYH2EURRYX7Q, seg_pos={0 0}, upload=335186]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 21 13:24:55.918121590 (381h55m55.789204635s) [noise=true , address=145.79.227.44:4195 , piece_id=QECQLX7KZVU7RMFVZ6TOHRIXR776DFZ4D44FKFNUWYH2EURRYX7Q, seg_pos={0 0}, upload=335186]: Long tail cancellation due to optimal threshold Dec 21 13:24:55.918156175 (381h55m55.789239221s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4CVCVL6PSTVUM33DZPLIU6QUN6JIXFCPBU6CNNA4P6YCHJ3PONA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:37888->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37888->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37888->45.91.116.13:4153: use of closed network connection Dec 21 13:24:55.918163128 (381h55m55.789246164s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4CVCVL6PSTVUM33DZPLIU6QUN6JIXFCPBU6CNNA4P6YCHJ3PONA, seg_pos={0 0}, upload=335186]: Long tail cancellation due to optimal threshold Dec 21 13:24:55.918263059 (381h55m55.789346094s) [seg_pos={0 0}, upload=335186]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 13:24:55.918332972 (381h55m55.789416017s) [upload=335186]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 13:24:55.918339284 (381h55m55.789422339s) [upload=335186]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 13:24:56.213644022 (381h55m56.084727067s) [upload=335186]: Done uploading segments... Dec 21 13:24:56.213663228 (381h55m56.084746263s) [upload=335186]: Upload finished. err: <nil> Duration: 7.505782433s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X....X........X.....................XXXX...X..X...........X.] duration: 5.877671199s trace: https://jaeger.dev.storj.io/trace/fa7962f52d9630d FAIL (2025-12-21 13:25:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: MN5IF2YLUACGZTO4BG2UQWBCD3MLHDBGHYSHO5PUYVVLODSL7H5A) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: JKUFZKN4XVAW7BIVYPOQJURRNEG7I444RJSZC5IGH2X5FFQOJ2OA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: L72GMUZ6FNO3ATO3OT2H2EZEUM3NATEGRKBSHUSGURSGETRV3BSA) context deadline exceeded ======LOGS====== gments... Dec 21 13:24:53.199397540 (381h55m53.070480575s) [upload=335186]: Got next segment. Inline: false Dec 21 13:24:53.199433087 (381h55m53.070516122s) [upload=335186]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 13:24:53.199437856 (381h55m53.070520901s) [upload=335186]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 13:24:53.446838205 (381h55m53.31792125s) [seg_pos={0 0}, upload=335186]: Begin upload segment... Dec 21 13:24:53.446894843 (381h55m53.317977878s) [seg_pos={0 0}, upload=335186]: Done begin upload segment. Dec 21 13:24:53.446931453 (381h55m53.318014488s) [noise=true , address=145.79.227.51:4409 , piece_id=F2B7K7BE7JLBZMIHP5STKVAN7MH2PCYKJ33ZYJF4XJMABZSI75YA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.446966390 (381h55m53.318049435s) [noise=true , address=145.79.227.43:4157 , piece_id=CUGHWACL4LKBK3K5R6AYAU7PZSWTR5LH2KDP6JVV36XMFJVAQRAA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.447612399 (381h55m53.318695444s) [noise=true , address=145.79.227.44:4195 , piece_id=QECQLX7KZVU7RMFVZ6TOHRIXR776DFZ4D44FKFNUWYH2EURRYX7Q, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.447623340 (381h55m53.318706395s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4CVCVL6PSTVUM33DZPLIU6QUN6JIXFCPBU6CNNA4P6YCHJ3PONA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.448102392 (381h55m53.319185437s) [noise=true , address=145.79.227.35:4481 , piece_id=BBT67CINYVQGAHO5U6KGRG6VDPVKUFQXMWWCFG362P5E2V3I26AA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.448110838 (381h55m53.319193893s) [noise=true , address=145.79.227.37:4566 , piece_id=N6IMZ662JFVQWWVACNZYIGNCLFDKA3PF74XBRMQVIQFTJYU6FWHQ, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.448491503 (381h55m53.319574538s) [noise=true , address=77.90.31.10:4517 , piece_id=JKUFZKN4XVAW7BIVYPOQJURRNEG7I444RJSZC5IGH2X5FFQOJ2OA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.448573760 (381h55m53.319656805s) [noise=true , address=77.90.31.17:4773 , piece_id=ZYGPOGT7746PEFLPVTO5YO54ZDMA4I7XDYKOUAQPDHM3N6Z6E7OA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.448884412 (381h55m53.319967447s) [noise=true , address=77.90.31.14:4663 , piece_id=OLWF23WAL6KNLJ3SJKVJZC3BDXE2GREL3FPRNQATRK5DJEOICLEA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.448945598 (381h55m53.320028643s) [noise=true , address=145.79.227.48:4302 , piece_id=WEI5KLMAZX4TDRDHDYQP3MFMJD7NUPVYDVWKWNGQWHI72XQTNYBQ, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.449274494 (381h55m53.320357529s) [noise=true , address=145.79.227.38:4979 , piece_id=WBXCJPW3DXG7ICOO5QUG26LMS7G73NJAYANQLXOTPZ2WA4DRKHYA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.449325291 (381h55m53.320408336s) [noise=true , address=145.79.227.15:4874 , piece_id=A6FUVEMRY45O5DX34PGHNMLFPKESXU4E7RFIBZUHOOMZ6ZCKZF7Q, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.449651352 (381h55m53.320734387s) [noise=true , address=145.79.227.45:4230 , piece_id=G5UHDEEHX7WF6HXAWC2PYFVUQUGYRQKDRUAEMDNRVHJY4C7MDIJA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.449709993 (381h55m53.320793028s) [noise=true , address=145.79.227.32:4834 , piece_id=C25VCQMG32RWVXI3ROQPPURSYE7XTB3RHTCAVZ7N5DTTKK3JEY4Q, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.450191931 (381h55m53.321274966s) [noise=true , address=45.91.116.6:4974 , piece_id=MN5IF2YLUACGZTO4BG2UQWBCD3MLHDBGHYSHO5PUYVVLODSL7H5A, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.451236851 (381h55m53.322319886s) [noise=true , address=45.91.116.8:4046 , piece_id=L72GMUZ6FNO3ATO3OT2H2EZEUM3NATEGRKBSHUSGURSGETRV3BSA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:55.053012716 (381h55m54.924095751s) [upload=335186]: Next returned nil segment Dec 21 13:24:55.053027694 (381h55m54.924110729s) [upload=335186]: Waiting for error group managing segments... Dec 21 13:24:55.437142389 (381h55m55.308225424s) [noise=true , address=145.79.227.43:4157 , piece_id=CUGHWACL4LKBK3K5R6AYAU7PZSWTR5LH2KDP6JVV36XMFJVAQRAA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.438255078 (381h55m55.309338113s) [noise=true , address=145.79.227.45:4230 , piece_id=G5UHDEEHX7WF6HXAWC2PYFVUQUGYRQKDRUAEMDNRVHJY4C7MDIJA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.439457849 (381h55m55.310540884s) [noise=true , address=145.79.227.37:4566 , piece_id=N6IMZ662JFVQWWVACNZYIGNCLFDKA3PF74XBRMQVIQFTJYU6FWHQ, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.440351931 (381h55m55.311434967s) [noise=true , address=145.79.227.32:4834 , piece_id=C25VCQMG32RWVXI3ROQPPURSYE7XTB3RHTCAVZ7N5DTTKK3JEY4Q, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.441163436 (381h55m55.312246481s) [noise=true , address=145.79.227.48:4302 , piece_id=WEI5KLMAZX4TDRDHDYQP3MFMJD7NUPVYDVWKWNGQWHI72XQTNYBQ, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.458667117 (381h55m55.329750162s) [noise=true , address=145.79.227.15:4874 , piece_id=A6FUVEMRY45O5DX34PGHNMLFPKESXU4E7RFIBZUHOOMZ6ZCKZF7Q, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.459123205 (381h55m55.33020624s) [noise=true , address=145.79.227.51:4409 , piece_id=F2B7K7BE7JLBZMIHP5STKVAN7MH2PCYKJ33ZYJF4XJMABZSI75YA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.461487727 (381h55m55.332570773s) [noise=true , address=145.79.227.35:4481 , piece_id=BBT67CINYVQGAHO5U6KGRG6VDPVKUFQXMWWCFG362P5E2V3I26AA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.465407611 (381h55m55.336490656s) [noise=true , address=145.79.227.38:4979 , piece_id=WBXCJPW3DXG7ICOO5QUG26LMS7G73NJAYANQLXOTPZ2WA4DRKHYA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.691010826 (381h55m55.562093861s) [noise=true , address=77.90.31.14:4663 , piece_id=OLWF23WAL6KNLJ3SJKVJZC3BDXE2GREL3FPRNQATRK5DJEOICLEA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.715824785 (381h55m55.58690783s) [noise=true , address=45.91.116.6:4974 , piece_id=MN5IF2YLUACGZTO4BG2UQWBCD3MLHDBGHYSHO5PUYVVLODSL7H5A, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.734305105 (381h55m55.60538815s) [noise=true , address=45.91.116.8:4046 , piece_id=L72GMUZ6FNO3ATO3OT2H2EZEUM3NATEGRKBSHUSGURSGETRV3BSA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.917719464 (381h55m55.788802499s) [noise=true , address=77.90.31.10:4517 , piece_id=JKUFZKN4XVAW7BIVYPOQJURRNEG7I444RJSZC5IGH2X5FFQOJ2OA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.917738280 (381h55m55.788821315s) [seg_pos={0 0}, upload=335186]: Segment reached optimal threshold of 13 pieces. Dec 21 13:24:55.917957757 (381h55m55.789040802s) [noise=true , address=77.90.31.17:4773 , piece_id=ZYGPOGT7746PEFLPVTO5YO54ZDMA4I7XDYKOUAQPDHM3N6Z6E7OA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:50438->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:50438->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50438->77.90.31.17:4773: use of closed network connection Dec 21 13:24:55.917964471 (381h55m55.789047506s) [noise=true , address=77.90.31.17:4773 , piece_id=ZYGPOGT7746PEFLPVTO5YO54ZDMA4I7XDYKOUAQPDHM3N6Z6E7OA, seg_pos={0 0}, upload=335186]: Long tail cancellation due to optimal threshold Dec 21 13:24:55.918111541 (381h55m55.789194586s) [noise=true , address=145.79.227.44:4195 , piece_id=QECQLX7KZVU7RMFVZ6TOHRIXR776DFZ4D44FKFNUWYH2EURRYX7Q, seg_pos={0 0}, upload=335186]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 21 13:24:55.918121590 (381h55m55.789204635s) [noise=true , address=145.79.227.44:4195 , piece_id=QECQLX7KZVU7RMFVZ6TOHRIXR776DFZ4D44FKFNUWYH2EURRYX7Q, seg_pos={0 0}, upload=335186]: Long tail cancellation due to optimal threshold Dec 21 13:24:55.918156175 (381h55m55.789239221s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4CVCVL6PSTVUM33DZPLIU6QUN6JIXFCPBU6CNNA4P6YCHJ3PONA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:37888->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37888->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37888->45.91.116.13:4153: use of closed network connection Dec 21 13:24:55.918163128 (381h55m55.789246164s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4CVCVL6PSTVUM33DZPLIU6QUN6JIXFCPBU6CNNA4P6YCHJ3PONA, seg_pos={0 0}, upload=335186]: Long tail cancellation due to optimal threshold Dec 21 13:24:55.918263059 (381h55m55.789346094s) [seg_pos={0 0}, upload=335186]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 13:24:55.918332972 (381h55m55.789416017s) [upload=335186]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 13:24:55.918339284 (381h55m55.789422339s) [upload=335186]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 13:24:56.213644022 (381h55m56.084727067s) [upload=335186]: Done uploading segments... Dec 21 13:24:56.213663228 (381h55m56.084746263s) [upload=335186]: Upload finished. err: <nil> Duration: 5.877701106s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.X....X........X.....................XXXX...X..X...........X] duration: 30.002259085s trace: https://jaeger.dev.storj.io/trace/fa7962f52d9630d FAIL (2025-12-21 13:25:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: MN5IF2YLUACGZTO4BG2UQWBCD3MLHDBGHYSHO5PUYVVLODSL7H5A) context deadline exceeded; piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: JKUFZKN4XVAW7BIVYPOQJURRNEG7I444RJSZC5IGH2X5FFQOJ2OA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: L72GMUZ6FNO3ATO3OT2H2EZEUM3NATEGRKBSHUSGURSGETRV3BSA) context deadline exceeded ======LOGS====== gments... Dec 21 13:24:53.199397540 (381h55m53.070480575s) [upload=335186]: Got next segment. Inline: false Dec 21 13:24:53.199433087 (381h55m53.070516122s) [upload=335186]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 13:24:53.199437856 (381h55m53.070520901s) [upload=335186]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 13:24:53.446838205 (381h55m53.31792125s) [seg_pos={0 0}, upload=335186]: Begin upload segment... Dec 21 13:24:53.446894843 (381h55m53.317977878s) [seg_pos={0 0}, upload=335186]: Done begin upload segment. Dec 21 13:24:53.446931453 (381h55m53.318014488s) [noise=true , address=145.79.227.51:4409 , piece_id=F2B7K7BE7JLBZMIHP5STKVAN7MH2PCYKJ33ZYJF4XJMABZSI75YA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.446966390 (381h55m53.318049435s) [noise=true , address=145.79.227.43:4157 , piece_id=CUGHWACL4LKBK3K5R6AYAU7PZSWTR5LH2KDP6JVV36XMFJVAQRAA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.447612399 (381h55m53.318695444s) [noise=true , address=145.79.227.44:4195 , piece_id=QECQLX7KZVU7RMFVZ6TOHRIXR776DFZ4D44FKFNUWYH2EURRYX7Q, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.447623340 (381h55m53.318706395s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4CVCVL6PSTVUM33DZPLIU6QUN6JIXFCPBU6CNNA4P6YCHJ3PONA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.448102392 (381h55m53.319185437s) [noise=true , address=145.79.227.35:4481 , piece_id=BBT67CINYVQGAHO5U6KGRG6VDPVKUFQXMWWCFG362P5E2V3I26AA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.448110838 (381h55m53.319193893s) [noise=true , address=145.79.227.37:4566 , piece_id=N6IMZ662JFVQWWVACNZYIGNCLFDKA3PF74XBRMQVIQFTJYU6FWHQ, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.448491503 (381h55m53.319574538s) [noise=true , address=77.90.31.10:4517 , piece_id=JKUFZKN4XVAW7BIVYPOQJURRNEG7I444RJSZC5IGH2X5FFQOJ2OA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.448573760 (381h55m53.319656805s) [noise=true , address=77.90.31.17:4773 , piece_id=ZYGPOGT7746PEFLPVTO5YO54ZDMA4I7XDYKOUAQPDHM3N6Z6E7OA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.448884412 (381h55m53.319967447s) [noise=true , address=77.90.31.14:4663 , piece_id=OLWF23WAL6KNLJ3SJKVJZC3BDXE2GREL3FPRNQATRK5DJEOICLEA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.448945598 (381h55m53.320028643s) [noise=true , address=145.79.227.48:4302 , piece_id=WEI5KLMAZX4TDRDHDYQP3MFMJD7NUPVYDVWKWNGQWHI72XQTNYBQ, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.449274494 (381h55m53.320357529s) [noise=true , address=145.79.227.38:4979 , piece_id=WBXCJPW3DXG7ICOO5QUG26LMS7G73NJAYANQLXOTPZ2WA4DRKHYA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.449325291 (381h55m53.320408336s) [noise=true , address=145.79.227.15:4874 , piece_id=A6FUVEMRY45O5DX34PGHNMLFPKESXU4E7RFIBZUHOOMZ6ZCKZF7Q, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.449651352 (381h55m53.320734387s) [noise=true , address=145.79.227.45:4230 , piece_id=G5UHDEEHX7WF6HXAWC2PYFVUQUGYRQKDRUAEMDNRVHJY4C7MDIJA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.449709993 (381h55m53.320793028s) [noise=true , address=145.79.227.32:4834 , piece_id=C25VCQMG32RWVXI3ROQPPURSYE7XTB3RHTCAVZ7N5DTTKK3JEY4Q, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.450191931 (381h55m53.321274966s) [noise=true , address=45.91.116.6:4974 , piece_id=MN5IF2YLUACGZTO4BG2UQWBCD3MLHDBGHYSHO5PUYVVLODSL7H5A, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:53.451236851 (381h55m53.322319886s) [noise=true , address=45.91.116.8:4046 , piece_id=L72GMUZ6FNO3ATO3OT2H2EZEUM3NATEGRKBSHUSGURSGETRV3BSA, seg_pos={0 0}, upload=335186]: Uploading piece... Dec 21 13:24:55.053012716 (381h55m54.924095751s) [upload=335186]: Next returned nil segment Dec 21 13:24:55.053027694 (381h55m54.924110729s) [upload=335186]: Waiting for error group managing segments... Dec 21 13:24:55.437142389 (381h55m55.308225424s) [noise=true , address=145.79.227.43:4157 , piece_id=CUGHWACL4LKBK3K5R6AYAU7PZSWTR5LH2KDP6JVV36XMFJVAQRAA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.438255078 (381h55m55.309338113s) [noise=true , address=145.79.227.45:4230 , piece_id=G5UHDEEHX7WF6HXAWC2PYFVUQUGYRQKDRUAEMDNRVHJY4C7MDIJA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.439457849 (381h55m55.310540884s) [noise=true , address=145.79.227.37:4566 , piece_id=N6IMZ662JFVQWWVACNZYIGNCLFDKA3PF74XBRMQVIQFTJYU6FWHQ, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.440351931 (381h55m55.311434967s) [noise=true , address=145.79.227.32:4834 , piece_id=C25VCQMG32RWVXI3ROQPPURSYE7XTB3RHTCAVZ7N5DTTKK3JEY4Q, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.441163436 (381h55m55.312246481s) [noise=true , address=145.79.227.48:4302 , piece_id=WEI5KLMAZX4TDRDHDYQP3MFMJD7NUPVYDVWKWNGQWHI72XQTNYBQ, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.458667117 (381h55m55.329750162s) [noise=true , address=145.79.227.15:4874 , piece_id=A6FUVEMRY45O5DX34PGHNMLFPKESXU4E7RFIBZUHOOMZ6ZCKZF7Q, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.459123205 (381h55m55.33020624s) [noise=true , address=145.79.227.51:4409 , piece_id=F2B7K7BE7JLBZMIHP5STKVAN7MH2PCYKJ33ZYJF4XJMABZSI75YA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.461487727 (381h55m55.332570773s) [noise=true , address=145.79.227.35:4481 , piece_id=BBT67CINYVQGAHO5U6KGRG6VDPVKUFQXMWWCFG362P5E2V3I26AA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.465407611 (381h55m55.336490656s) [noise=true , address=145.79.227.38:4979 , piece_id=WBXCJPW3DXG7ICOO5QUG26LMS7G73NJAYANQLXOTPZ2WA4DRKHYA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.691010826 (381h55m55.562093861s) [noise=true , address=77.90.31.14:4663 , piece_id=OLWF23WAL6KNLJ3SJKVJZC3BDXE2GREL3FPRNQATRK5DJEOICLEA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.715824785 (381h55m55.58690783s) [noise=true , address=45.91.116.6:4974 , piece_id=MN5IF2YLUACGZTO4BG2UQWBCD3MLHDBGHYSHO5PUYVVLODSL7H5A, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.734305105 (381h55m55.60538815s) [noise=true , address=45.91.116.8:4046 , piece_id=L72GMUZ6FNO3ATO3OT2H2EZEUM3NATEGRKBSHUSGURSGETRV3BSA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.917719464 (381h55m55.788802499s) [noise=true , address=77.90.31.10:4517 , piece_id=JKUFZKN4XVAW7BIVYPOQJURRNEG7I444RJSZC5IGH2X5FFQOJ2OA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: <nil> Dec 21 13:24:55.917738280 (381h55m55.788821315s) [seg_pos={0 0}, upload=335186]: Segment reached optimal threshold of 13 pieces. Dec 21 13:24:55.917957757 (381h55m55.789040802s) [noise=true , address=77.90.31.17:4773 , piece_id=ZYGPOGT7746PEFLPVTO5YO54ZDMA4I7XDYKOUAQPDHM3N6Z6E7OA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:50438->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:50438->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50438->77.90.31.17:4773: use of closed network connection Dec 21 13:24:55.917964471 (381h55m55.789047506s) [noise=true , address=77.90.31.17:4773 , piece_id=ZYGPOGT7746PEFLPVTO5YO54ZDMA4I7XDYKOUAQPDHM3N6Z6E7OA, seg_pos={0 0}, upload=335186]: Long tail cancellation due to optimal threshold Dec 21 13:24:55.918111541 (381h55m55.789194586s) [noise=true , address=145.79.227.44:4195 , piece_id=QECQLX7KZVU7RMFVZ6TOHRIXR776DFZ4D44FKFNUWYH2EURRYX7Q, seg_pos={0 0}, upload=335186]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 21 13:24:55.918121590 (381h55m55.789204635s) [noise=true , address=145.79.227.44:4195 , piece_id=QECQLX7KZVU7RMFVZ6TOHRIXR776DFZ4D44FKFNUWYH2EURRYX7Q, seg_pos={0 0}, upload=335186]: Long tail cancellation due to optimal threshold Dec 21 13:24:55.918156175 (381h55m55.789239221s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4CVCVL6PSTVUM33DZPLIU6QUN6JIXFCPBU6CNNA4P6YCHJ3PONA, seg_pos={0 0}, upload=335186]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:37888->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:37888->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37888->45.91.116.13:4153: use of closed network connection Dec 21 13:24:55.918163128 (381h55m55.789246164s) [noise=true , address=45.91.116.13:4153 , piece_id=Y4CVCVL6PSTVUM33DZPLIU6QUN6JIXFCPBU6CNNA4P6YCHJ3PONA, seg_pos={0 0}, upload=335186]: Long tail cancellation due to optimal threshold Dec 21 13:24:55.918263059 (381h55m55.789346094s) [seg_pos={0 0}, upload=335186]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 13:24:55.918332972 (381h55m55.789416017s) [upload=335186]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 13:24:55.918339284 (381h55m55.789422339s) [upload=335186]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 13:24:56.213644022 (381h55m56.084727067s) [upload=335186]: Done uploading segments... Dec 21 13:24:56.213663228 (381h55m56.084746263s) [upload=335186]: Upload finished. err: <nil> Duration: 30.002278302s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..X....X........X.....................XXXX...X..X...........] duration: 5.766093938s trace: https://jaeger.dev.storj.io/trace/2d9809728bd4cd0c FAIL (2025-12-21 13:24:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: GMCGUAPW66BGV34OH62E5Q66I27LMCCJSGYR5TUWJURFP77KVGJA) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: HMF7YEOHQ5GMCTBMEW3LBCDU2OEUL37IHAHKJTWFID2OUNULAZXQ) context deadline exceeded ======LOGS====== Dec 21 13:15:56.527259459 (381h46m56.398342504s) [upload=335056]: Starting upload Dec 21 13:15:56.527348678 (381h46m56.398431733s) [upload=335056]: Done starting upload Dec 21 13:15:56.527441445 (381h46m56.39852448s) [upload=335056]: Uploading segments... Dec 21 13:15:56.527575821 (381h46m56.398658856s) [upload=335056]: Got next segment. Inline: false Dec 21 13:15:56.528555026 (381h46m56.399638061s) [upload=335056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 13:15:56.528563231 (381h46m56.399646266s) [upload=335056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 13:15:56.775777978 (381h46m56.646861033s) [seg_pos={0 0}, upload=335056]: Begin upload segment... Dec 21 13:15:56.775835256 (381h46m56.646918291s) [seg_pos={0 0}, upload=335056]: Done begin upload segment. Dec 21 13:15:56.775875322 (381h46m56.646958378s) [noise=true , address=145.79.227.43:4157 , piece_id=FUCMIO5FINNCMGYXMDL3G7GY3H7XYQMXSV65F6ILKVWFDIK3BK6Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.775888899 (381h46m56.646971934s) [noise=true , address=45.91.116.14:4189 , piece_id=HMF7YEOHQ5GMCTBMEW3LBCDU2OEUL37IHAHKJTWFID2OUNULAZXQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.776387338 (381h46m56.647470383s) [noise=true , address=145.79.227.48:4301 , piece_id=LSMT6PC7K6BUFGXFFCNLN5TKMB4F7ARL24QRUGLPOO6RP5RNP7YA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.776605343 (381h46m56.647688378s) [noise=true , address=145.79.227.50:4375 , piece_id=U4WW6N3OB7G5T5BZXBYLFWMFERKOEA2VRF7JHRFMNZHKQRVF7U7Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.776799853 (381h46m56.647882898s) [noise=true , address=45.91.116.8:4046 , piece_id=5WFCBJKXGZX63HORQJDVDBDG46BMZ3VE7WWYZPEKXLWUKQ434RYQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777123059 (381h46m56.648206094s) [noise=true , address=145.79.227.46:4603 , piece_id=IBB2HKDX4UOF4OBD6OCPAELN7K5KRKBB3TNCAD4AOAAM3DWT6BDA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777207400 (381h46m56.648290445s) [noise=true , address=145.79.227.52:4445 , piece_id=H7BWJDSSISRYQGOBM5ZKVHR4FKMH5LTONG2T56WTX6IKH4IGIGQQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777533130 (381h46m56.648616165s) [noise=true , address=145.79.227.37:4565 , piece_id=GU35UYW7DUSW2QI7DM4H7GUILILZN2BGPKR5SSPAYC7MTFYI5I4Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777602983 (381h46m56.648686018s) [noise=true , address=45.91.116.13:4156 , piece_id=GMCGUAPW66BGV34OH62E5Q66I27LMCCJSGYR5TUWJURFP77KVGJA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777919996 (381h46m56.649003111s) [noise=true , address=145.79.227.38:4979 , piece_id=A3SZK4HNWGB3AGYKDXVT57CAP5QPEXCJ2OZHIC7XEOEQPNHPV77A, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778138662 (381h46m56.649221697s) [noise=true , address=145.79.227.8:4330 , piece_id=O3BAAICZCQULVD6TSBPOGEHC7FNAB53QNL26OYVX5ED6ESWU64PQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778531932 (381h46m56.649614967s) [noise=true , address=45.91.116.10:4084 , piece_id=7X5ACNCEMTSKIVIDJQ72IPMJGSFC47WO6VN5JANFEMYIUDUZ6ALQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778556999 (381h46m56.649640044s) [noise=true , address=145.79.227.39:4013 , piece_id=GP6DMTAQHDHF2JTUVD6NWPWCS4F3LFMSDGKFK2TUTWQ7U6MXXB5Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778969454 (381h46m56.650052499s) [noise=true , address=77.90.31.10:4519 , piece_id=YWYV2OZRPNYCRFZUXU7I4NPSDZ4TPBPDKZLZTFDP3Z2T6VBD77HA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.779064235 (381h46m56.65014728s) [noise=true , address=145.79.227.29:4943 , piece_id=AR2GVI4J2Q5JY7JE7G7ZYFRNYTW6HHAJEA5AUJTUNEBL6SYEUVUA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.779624251 (381h46m56.650707296s) [noise=true , address=145.79.227.45:4230 , piece_id=WOKT7BUVBTCQMPKTNBWI5IBEVVBCGLJSWVBXPD3DSRXWXS5ZODAA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:58.473831251 (381h46m58.344914296s) [upload=335056]: Next returned nil segment Dec 21 13:15:58.473877700 (381h46m58.344960735s) [upload=335056]: Waiting for error group managing segments... Dec 21 13:15:58.750808791 (381h46m58.621891826s) [noise=true , address=145.79.227.8:4330 , piece_id=O3BAAICZCQULVD6TSBPOGEHC7FNAB53QNL26OYVX5ED6ESWU64PQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.760307128 (381h46m58.631390163s) [noise=true , address=145.79.227.37:4565 , piece_id=GU35UYW7DUSW2QI7DM4H7GUILILZN2BGPKR5SSPAYC7MTFYI5I4Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.761544894 (381h46m58.632627929s) [noise=true , address=145.79.227.50:4375 , piece_id=U4WW6N3OB7G5T5BZXBYLFWMFERKOEA2VRF7JHRFMNZHKQRVF7U7Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.767805456 (381h46m58.638888501s) [noise=true , address=145.79.227.38:4979 , piece_id=A3SZK4HNWGB3AGYKDXVT57CAP5QPEXCJ2OZHIC7XEOEQPNHPV77A, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.770873438 (381h46m58.641956473s) [noise=true , address=145.79.227.48:4301 , piece_id=LSMT6PC7K6BUFGXFFCNLN5TKMB4F7ARL24QRUGLPOO6RP5RNP7YA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.771314767 (381h46m58.642397802s) [noise=true , address=145.79.227.43:4157 , piece_id=FUCMIO5FINNCMGYXMDL3G7GY3H7XYQMXSV65F6ILKVWFDIK3BK6Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.792102381 (381h46m58.663185416s) [noise=true , address=145.79.227.46:4603 , piece_id=IBB2HKDX4UOF4OBD6OCPAELN7K5KRKBB3TNCAD4AOAAM3DWT6BDA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.800571156 (381h46m58.671654191s) [noise=true , address=145.79.227.29:4943 , piece_id=AR2GVI4J2Q5JY7JE7G7ZYFRNYTW6HHAJEA5AUJTUNEBL6SYEUVUA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.827627135 (381h46m58.69871018s) [noise=true , address=145.79.227.52:4445 , piece_id=H7BWJDSSISRYQGOBM5ZKVHR4FKMH5LTONG2T56WTX6IKH4IGIGQQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.832837256 (381h46m58.703920291s) [noise=true , address=145.79.227.45:4230 , piece_id=WOKT7BUVBTCQMPKTNBWI5IBEVVBCGLJSWVBXPD3DSRXWXS5ZODAA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.856094943 (381h46m58.727177988s) [noise=true , address=145.79.227.39:4013 , piece_id=GP6DMTAQHDHF2JTUVD6NWPWCS4F3LFMSDGKFK2TUTWQ7U6MXXB5Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:59.008546104 (381h46m58.879629149s) [noise=true , address=45.91.116.13:4156 , piece_id=GMCGUAPW66BGV34OH62E5Q66I27LMCCJSGYR5TUWJURFP77KVGJA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:59.052840027 (381h46m58.923923062s) [noise=true , address=45.91.116.14:4189 , piece_id=HMF7YEOHQ5GMCTBMEW3LBCDU2OEUL37IHAHKJTWFID2OUNULAZXQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:59.052858953 (381h46m58.923941988s) [seg_pos={0 0}, upload=335056]: Segment reached optimal threshold of 13 pieces. Dec 21 13:15:59.053141110 (381h46m58.924224155s) [noise=true , address=45.91.116.10:4084 , piece_id=7X5ACNCEMTSKIVIDJQ72IPMJGSFC47WO6VN5JANFEMYIUDUZ6ALQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 21 13:15:59.053148975 (381h46m58.92423202s) [noise=true , address=45.91.116.10:4084 , piece_id=7X5ACNCEMTSKIVIDJQ72IPMJGSFC47WO6VN5JANFEMYIUDUZ6ALQ, seg_pos={0 0}, upload=335056]: Long tail cancellation due to optimal threshold Dec 21 13:15:59.053223627 (381h46m58.924306662s) [noise=true , address=45.91.116.8:4046 , piece_id=5WFCBJKXGZX63HORQJDVDBDG46BMZ3VE7WWYZPEKXLWUKQ434RYQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 21 13:15:59.053253414 (381h46m58.924336449s) [noise=true , address=45.91.116.8:4046 , piece_id=5WFCBJKXGZX63HORQJDVDBDG46BMZ3VE7WWYZPEKXLWUKQ434RYQ, seg_pos={0 0}, upload=335056]: Long tail cancellation due to optimal threshold Dec 21 13:15:59.053407067 (381h46m58.924490112s) [noise=true , address=77.90.31.10:4519 , piece_id=YWYV2OZRPNYCRFZUXU7I4NPSDZ4TPBPDKZLZTFDP3Z2T6VBD77HA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 21 13:15:59.053414541 (381h46m58.924497576s) [noise=true , address=77.90.31.10:4519 , piece_id=YWYV2OZRPNYCRFZUXU7I4NPSDZ4TPBPDKZLZTFDP3Z2T6VBD77HA, seg_pos={0 0}, upload=335056]: Long tail cancellation due to optimal threshold Dec 21 13:15:59.053429720 (381h46m58.924512765s) [seg_pos={0 0}, upload=335056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 13:15:59.053500063 (381h46m58.924583109s) [upload=335056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 13:15:59.053507238 (381h46m58.924590283s) [upload=335056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 13:15:59.348061896 (381h46m59.219144941s) [upload=335056]: Done uploading segments... Dec 21 13:15:59.348081795 (381h46m59.21916483s) [upload=335056]: Upload finished. err: <nil> Duration: 5.766125558s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X..X....X........X.....................XXXX...X..X..........] duration: 5.899378312s trace: https://jaeger.dev.storj.io/trace/2d9809728bd4cd0c FAIL (2025-12-21 13:23:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: GMCGUAPW66BGV34OH62E5Q66I27LMCCJSGYR5TUWJURFP77KVGJA) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: HMF7YEOHQ5GMCTBMEW3LBCDU2OEUL37IHAHKJTWFID2OUNULAZXQ) context deadline exceeded ======LOGS====== Dec 21 13:15:56.527259459 (381h46m56.398342504s) [upload=335056]: Starting upload Dec 21 13:15:56.527348678 (381h46m56.398431733s) [upload=335056]: Done starting upload Dec 21 13:15:56.527441445 (381h46m56.39852448s) [upload=335056]: Uploading segments... Dec 21 13:15:56.527575821 (381h46m56.398658856s) [upload=335056]: Got next segment. Inline: false Dec 21 13:15:56.528555026 (381h46m56.399638061s) [upload=335056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 13:15:56.528563231 (381h46m56.399646266s) [upload=335056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 13:15:56.775777978 (381h46m56.646861033s) [seg_pos={0 0}, upload=335056]: Begin upload segment... Dec 21 13:15:56.775835256 (381h46m56.646918291s) [seg_pos={0 0}, upload=335056]: Done begin upload segment. Dec 21 13:15:56.775875322 (381h46m56.646958378s) [noise=true , address=145.79.227.43:4157 , piece_id=FUCMIO5FINNCMGYXMDL3G7GY3H7XYQMXSV65F6ILKVWFDIK3BK6Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.775888899 (381h46m56.646971934s) [noise=true , address=45.91.116.14:4189 , piece_id=HMF7YEOHQ5GMCTBMEW3LBCDU2OEUL37IHAHKJTWFID2OUNULAZXQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.776387338 (381h46m56.647470383s) [noise=true , address=145.79.227.48:4301 , piece_id=LSMT6PC7K6BUFGXFFCNLN5TKMB4F7ARL24QRUGLPOO6RP5RNP7YA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.776605343 (381h46m56.647688378s) [noise=true , address=145.79.227.50:4375 , piece_id=U4WW6N3OB7G5T5BZXBYLFWMFERKOEA2VRF7JHRFMNZHKQRVF7U7Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.776799853 (381h46m56.647882898s) [noise=true , address=45.91.116.8:4046 , piece_id=5WFCBJKXGZX63HORQJDVDBDG46BMZ3VE7WWYZPEKXLWUKQ434RYQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777123059 (381h46m56.648206094s) [noise=true , address=145.79.227.46:4603 , piece_id=IBB2HKDX4UOF4OBD6OCPAELN7K5KRKBB3TNCAD4AOAAM3DWT6BDA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777207400 (381h46m56.648290445s) [noise=true , address=145.79.227.52:4445 , piece_id=H7BWJDSSISRYQGOBM5ZKVHR4FKMH5LTONG2T56WTX6IKH4IGIGQQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777533130 (381h46m56.648616165s) [noise=true , address=145.79.227.37:4565 , piece_id=GU35UYW7DUSW2QI7DM4H7GUILILZN2BGPKR5SSPAYC7MTFYI5I4Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777602983 (381h46m56.648686018s) [noise=true , address=45.91.116.13:4156 , piece_id=GMCGUAPW66BGV34OH62E5Q66I27LMCCJSGYR5TUWJURFP77KVGJA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777919996 (381h46m56.649003111s) [noise=true , address=145.79.227.38:4979 , piece_id=A3SZK4HNWGB3AGYKDXVT57CAP5QPEXCJ2OZHIC7XEOEQPNHPV77A, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778138662 (381h46m56.649221697s) [noise=true , address=145.79.227.8:4330 , piece_id=O3BAAICZCQULVD6TSBPOGEHC7FNAB53QNL26OYVX5ED6ESWU64PQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778531932 (381h46m56.649614967s) [noise=true , address=45.91.116.10:4084 , piece_id=7X5ACNCEMTSKIVIDJQ72IPMJGSFC47WO6VN5JANFEMYIUDUZ6ALQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778556999 (381h46m56.649640044s) [noise=true , address=145.79.227.39:4013 , piece_id=GP6DMTAQHDHF2JTUVD6NWPWCS4F3LFMSDGKFK2TUTWQ7U6MXXB5Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778969454 (381h46m56.650052499s) [noise=true , address=77.90.31.10:4519 , piece_id=YWYV2OZRPNYCRFZUXU7I4NPSDZ4TPBPDKZLZTFDP3Z2T6VBD77HA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.779064235 (381h46m56.65014728s) [noise=true , address=145.79.227.29:4943 , piece_id=AR2GVI4J2Q5JY7JE7G7ZYFRNYTW6HHAJEA5AUJTUNEBL6SYEUVUA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.779624251 (381h46m56.650707296s) [noise=true , address=145.79.227.45:4230 , piece_id=WOKT7BUVBTCQMPKTNBWI5IBEVVBCGLJSWVBXPD3DSRXWXS5ZODAA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:58.473831251 (381h46m58.344914296s) [upload=335056]: Next returned nil segment Dec 21 13:15:58.473877700 (381h46m58.344960735s) [upload=335056]: Waiting for error group managing segments... Dec 21 13:15:58.750808791 (381h46m58.621891826s) [noise=true , address=145.79.227.8:4330 , piece_id=O3BAAICZCQULVD6TSBPOGEHC7FNAB53QNL26OYVX5ED6ESWU64PQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.760307128 (381h46m58.631390163s) [noise=true , address=145.79.227.37:4565 , piece_id=GU35UYW7DUSW2QI7DM4H7GUILILZN2BGPKR5SSPAYC7MTFYI5I4Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.761544894 (381h46m58.632627929s) [noise=true , address=145.79.227.50:4375 , piece_id=U4WW6N3OB7G5T5BZXBYLFWMFERKOEA2VRF7JHRFMNZHKQRVF7U7Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.767805456 (381h46m58.638888501s) [noise=true , address=145.79.227.38:4979 , piece_id=A3SZK4HNWGB3AGYKDXVT57CAP5QPEXCJ2OZHIC7XEOEQPNHPV77A, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.770873438 (381h46m58.641956473s) [noise=true , address=145.79.227.48:4301 , piece_id=LSMT6PC7K6BUFGXFFCNLN5TKMB4F7ARL24QRUGLPOO6RP5RNP7YA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.771314767 (381h46m58.642397802s) [noise=true , address=145.79.227.43:4157 , piece_id=FUCMIO5FINNCMGYXMDL3G7GY3H7XYQMXSV65F6ILKVWFDIK3BK6Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.792102381 (381h46m58.663185416s) [noise=true , address=145.79.227.46:4603 , piece_id=IBB2HKDX4UOF4OBD6OCPAELN7K5KRKBB3TNCAD4AOAAM3DWT6BDA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.800571156 (381h46m58.671654191s) [noise=true , address=145.79.227.29:4943 , piece_id=AR2GVI4J2Q5JY7JE7G7ZYFRNYTW6HHAJEA5AUJTUNEBL6SYEUVUA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.827627135 (381h46m58.69871018s) [noise=true , address=145.79.227.52:4445 , piece_id=H7BWJDSSISRYQGOBM5ZKVHR4FKMH5LTONG2T56WTX6IKH4IGIGQQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.832837256 (381h46m58.703920291s) [noise=true , address=145.79.227.45:4230 , piece_id=WOKT7BUVBTCQMPKTNBWI5IBEVVBCGLJSWVBXPD3DSRXWXS5ZODAA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.856094943 (381h46m58.727177988s) [noise=true , address=145.79.227.39:4013 , piece_id=GP6DMTAQHDHF2JTUVD6NWPWCS4F3LFMSDGKFK2TUTWQ7U6MXXB5Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:59.008546104 (381h46m58.879629149s) [noise=true , address=45.91.116.13:4156 , piece_id=GMCGUAPW66BGV34OH62E5Q66I27LMCCJSGYR5TUWJURFP77KVGJA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:59.052840027 (381h46m58.923923062s) [noise=true , address=45.91.116.14:4189 , piece_id=HMF7YEOHQ5GMCTBMEW3LBCDU2OEUL37IHAHKJTWFID2OUNULAZXQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:59.052858953 (381h46m58.923941988s) [seg_pos={0 0}, upload=335056]: Segment reached optimal threshold of 13 pieces. Dec 21 13:15:59.053141110 (381h46m58.924224155s) [noise=true , address=45.91.116.10:4084 , piece_id=7X5ACNCEMTSKIVIDJQ72IPMJGSFC47WO6VN5JANFEMYIUDUZ6ALQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 21 13:15:59.053148975 (381h46m58.92423202s) [noise=true , address=45.91.116.10:4084 , piece_id=7X5ACNCEMTSKIVIDJQ72IPMJGSFC47WO6VN5JANFEMYIUDUZ6ALQ, seg_pos={0 0}, upload=335056]: Long tail cancellation due to optimal threshold Dec 21 13:15:59.053223627 (381h46m58.924306662s) [noise=true , address=45.91.116.8:4046 , piece_id=5WFCBJKXGZX63HORQJDVDBDG46BMZ3VE7WWYZPEKXLWUKQ434RYQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 21 13:15:59.053253414 (381h46m58.924336449s) [noise=true , address=45.91.116.8:4046 , piece_id=5WFCBJKXGZX63HORQJDVDBDG46BMZ3VE7WWYZPEKXLWUKQ434RYQ, seg_pos={0 0}, upload=335056]: Long tail cancellation due to optimal threshold Dec 21 13:15:59.053407067 (381h46m58.924490112s) [noise=true , address=77.90.31.10:4519 , piece_id=YWYV2OZRPNYCRFZUXU7I4NPSDZ4TPBPDKZLZTFDP3Z2T6VBD77HA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 21 13:15:59.053414541 (381h46m58.924497576s) [noise=true , address=77.90.31.10:4519 , piece_id=YWYV2OZRPNYCRFZUXU7I4NPSDZ4TPBPDKZLZTFDP3Z2T6VBD77HA, seg_pos={0 0}, upload=335056]: Long tail cancellation due to optimal threshold Dec 21 13:15:59.053429720 (381h46m58.924512765s) [seg_pos={0 0}, upload=335056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 13:15:59.053500063 (381h46m58.924583109s) [upload=335056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 13:15:59.053507238 (381h46m58.924590283s) [upload=335056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 13:15:59.348061896 (381h46m59.219144941s) [upload=335056]: Done uploading segments... Dec 21 13:15:59.348081795 (381h46m59.21916483s) [upload=335056]: Upload finished. err: <nil> Duration: 5.899396698s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.X..X....X........X.....................XXXX...X..X.........] duration: 5.763662113s trace: https://jaeger.dev.storj.io/trace/2d9809728bd4cd0c FAIL (2025-12-21 13:22:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: GMCGUAPW66BGV34OH62E5Q66I27LMCCJSGYR5TUWJURFP77KVGJA) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: HMF7YEOHQ5GMCTBMEW3LBCDU2OEUL37IHAHKJTWFID2OUNULAZXQ) context deadline exceeded ======LOGS====== Dec 21 13:15:56.527259459 (381h46m56.398342504s) [upload=335056]: Starting upload Dec 21 13:15:56.527348678 (381h46m56.398431733s) [upload=335056]: Done starting upload Dec 21 13:15:56.527441445 (381h46m56.39852448s) [upload=335056]: Uploading segments... Dec 21 13:15:56.527575821 (381h46m56.398658856s) [upload=335056]: Got next segment. Inline: false Dec 21 13:15:56.528555026 (381h46m56.399638061s) [upload=335056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 13:15:56.528563231 (381h46m56.399646266s) [upload=335056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 13:15:56.775777978 (381h46m56.646861033s) [seg_pos={0 0}, upload=335056]: Begin upload segment... Dec 21 13:15:56.775835256 (381h46m56.646918291s) [seg_pos={0 0}, upload=335056]: Done begin upload segment. Dec 21 13:15:56.775875322 (381h46m56.646958378s) [noise=true , address=145.79.227.43:4157 , piece_id=FUCMIO5FINNCMGYXMDL3G7GY3H7XYQMXSV65F6ILKVWFDIK3BK6Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.775888899 (381h46m56.646971934s) [noise=true , address=45.91.116.14:4189 , piece_id=HMF7YEOHQ5GMCTBMEW3LBCDU2OEUL37IHAHKJTWFID2OUNULAZXQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.776387338 (381h46m56.647470383s) [noise=true , address=145.79.227.48:4301 , piece_id=LSMT6PC7K6BUFGXFFCNLN5TKMB4F7ARL24QRUGLPOO6RP5RNP7YA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.776605343 (381h46m56.647688378s) [noise=true , address=145.79.227.50:4375 , piece_id=U4WW6N3OB7G5T5BZXBYLFWMFERKOEA2VRF7JHRFMNZHKQRVF7U7Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.776799853 (381h46m56.647882898s) [noise=true , address=45.91.116.8:4046 , piece_id=5WFCBJKXGZX63HORQJDVDBDG46BMZ3VE7WWYZPEKXLWUKQ434RYQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777123059 (381h46m56.648206094s) [noise=true , address=145.79.227.46:4603 , piece_id=IBB2HKDX4UOF4OBD6OCPAELN7K5KRKBB3TNCAD4AOAAM3DWT6BDA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777207400 (381h46m56.648290445s) [noise=true , address=145.79.227.52:4445 , piece_id=H7BWJDSSISRYQGOBM5ZKVHR4FKMH5LTONG2T56WTX6IKH4IGIGQQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777533130 (381h46m56.648616165s) [noise=true , address=145.79.227.37:4565 , piece_id=GU35UYW7DUSW2QI7DM4H7GUILILZN2BGPKR5SSPAYC7MTFYI5I4Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777602983 (381h46m56.648686018s) [noise=true , address=45.91.116.13:4156 , piece_id=GMCGUAPW66BGV34OH62E5Q66I27LMCCJSGYR5TUWJURFP77KVGJA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777919996 (381h46m56.649003111s) [noise=true , address=145.79.227.38:4979 , piece_id=A3SZK4HNWGB3AGYKDXVT57CAP5QPEXCJ2OZHIC7XEOEQPNHPV77A, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778138662 (381h46m56.649221697s) [noise=true , address=145.79.227.8:4330 , piece_id=O3BAAICZCQULVD6TSBPOGEHC7FNAB53QNL26OYVX5ED6ESWU64PQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778531932 (381h46m56.649614967s) [noise=true , address=45.91.116.10:4084 , piece_id=7X5ACNCEMTSKIVIDJQ72IPMJGSFC47WO6VN5JANFEMYIUDUZ6ALQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778556999 (381h46m56.649640044s) [noise=true , address=145.79.227.39:4013 , piece_id=GP6DMTAQHDHF2JTUVD6NWPWCS4F3LFMSDGKFK2TUTWQ7U6MXXB5Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778969454 (381h46m56.650052499s) [noise=true , address=77.90.31.10:4519 , piece_id=YWYV2OZRPNYCRFZUXU7I4NPSDZ4TPBPDKZLZTFDP3Z2T6VBD77HA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.779064235 (381h46m56.65014728s) [noise=true , address=145.79.227.29:4943 , piece_id=AR2GVI4J2Q5JY7JE7G7ZYFRNYTW6HHAJEA5AUJTUNEBL6SYEUVUA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.779624251 (381h46m56.650707296s) [noise=true , address=145.79.227.45:4230 , piece_id=WOKT7BUVBTCQMPKTNBWI5IBEVVBCGLJSWVBXPD3DSRXWXS5ZODAA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:58.473831251 (381h46m58.344914296s) [upload=335056]: Next returned nil segment Dec 21 13:15:58.473877700 (381h46m58.344960735s) [upload=335056]: Waiting for error group managing segments... Dec 21 13:15:58.750808791 (381h46m58.621891826s) [noise=true , address=145.79.227.8:4330 , piece_id=O3BAAICZCQULVD6TSBPOGEHC7FNAB53QNL26OYVX5ED6ESWU64PQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.760307128 (381h46m58.631390163s) [noise=true , address=145.79.227.37:4565 , piece_id=GU35UYW7DUSW2QI7DM4H7GUILILZN2BGPKR5SSPAYC7MTFYI5I4Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.761544894 (381h46m58.632627929s) [noise=true , address=145.79.227.50:4375 , piece_id=U4WW6N3OB7G5T5BZXBYLFWMFERKOEA2VRF7JHRFMNZHKQRVF7U7Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.767805456 (381h46m58.638888501s) [noise=true , address=145.79.227.38:4979 , piece_id=A3SZK4HNWGB3AGYKDXVT57CAP5QPEXCJ2OZHIC7XEOEQPNHPV77A, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.770873438 (381h46m58.641956473s) [noise=true , address=145.79.227.48:4301 , piece_id=LSMT6PC7K6BUFGXFFCNLN5TKMB4F7ARL24QRUGLPOO6RP5RNP7YA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.771314767 (381h46m58.642397802s) [noise=true , address=145.79.227.43:4157 , piece_id=FUCMIO5FINNCMGYXMDL3G7GY3H7XYQMXSV65F6ILKVWFDIK3BK6Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.792102381 (381h46m58.663185416s) [noise=true , address=145.79.227.46:4603 , piece_id=IBB2HKDX4UOF4OBD6OCPAELN7K5KRKBB3TNCAD4AOAAM3DWT6BDA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.800571156 (381h46m58.671654191s) [noise=true , address=145.79.227.29:4943 , piece_id=AR2GVI4J2Q5JY7JE7G7ZYFRNYTW6HHAJEA5AUJTUNEBL6SYEUVUA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.827627135 (381h46m58.69871018s) [noise=true , address=145.79.227.52:4445 , piece_id=H7BWJDSSISRYQGOBM5ZKVHR4FKMH5LTONG2T56WTX6IKH4IGIGQQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.832837256 (381h46m58.703920291s) [noise=true , address=145.79.227.45:4230 , piece_id=WOKT7BUVBTCQMPKTNBWI5IBEVVBCGLJSWVBXPD3DSRXWXS5ZODAA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.856094943 (381h46m58.727177988s) [noise=true , address=145.79.227.39:4013 , piece_id=GP6DMTAQHDHF2JTUVD6NWPWCS4F3LFMSDGKFK2TUTWQ7U6MXXB5Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:59.008546104 (381h46m58.879629149s) [noise=true , address=45.91.116.13:4156 , piece_id=GMCGUAPW66BGV34OH62E5Q66I27LMCCJSGYR5TUWJURFP77KVGJA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:59.052840027 (381h46m58.923923062s) [noise=true , address=45.91.116.14:4189 , piece_id=HMF7YEOHQ5GMCTBMEW3LBCDU2OEUL37IHAHKJTWFID2OUNULAZXQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:59.052858953 (381h46m58.923941988s) [seg_pos={0 0}, upload=335056]: Segment reached optimal threshold of 13 pieces. Dec 21 13:15:59.053141110 (381h46m58.924224155s) [noise=true , address=45.91.116.10:4084 , piece_id=7X5ACNCEMTSKIVIDJQ72IPMJGSFC47WO6VN5JANFEMYIUDUZ6ALQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 21 13:15:59.053148975 (381h46m58.92423202s) [noise=true , address=45.91.116.10:4084 , piece_id=7X5ACNCEMTSKIVIDJQ72IPMJGSFC47WO6VN5JANFEMYIUDUZ6ALQ, seg_pos={0 0}, upload=335056]: Long tail cancellation due to optimal threshold Dec 21 13:15:59.053223627 (381h46m58.924306662s) [noise=true , address=45.91.116.8:4046 , piece_id=5WFCBJKXGZX63HORQJDVDBDG46BMZ3VE7WWYZPEKXLWUKQ434RYQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 21 13:15:59.053253414 (381h46m58.924336449s) [noise=true , address=45.91.116.8:4046 , piece_id=5WFCBJKXGZX63HORQJDVDBDG46BMZ3VE7WWYZPEKXLWUKQ434RYQ, seg_pos={0 0}, upload=335056]: Long tail cancellation due to optimal threshold Dec 21 13:15:59.053407067 (381h46m58.924490112s) [noise=true , address=77.90.31.10:4519 , piece_id=YWYV2OZRPNYCRFZUXU7I4NPSDZ4TPBPDKZLZTFDP3Z2T6VBD77HA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 21 13:15:59.053414541 (381h46m58.924497576s) [noise=true , address=77.90.31.10:4519 , piece_id=YWYV2OZRPNYCRFZUXU7I4NPSDZ4TPBPDKZLZTFDP3Z2T6VBD77HA, seg_pos={0 0}, upload=335056]: Long tail cancellation due to optimal threshold Dec 21 13:15:59.053429720 (381h46m58.924512765s) [seg_pos={0 0}, upload=335056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 13:15:59.053500063 (381h46m58.924583109s) [upload=335056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 13:15:59.053507238 (381h46m58.924590283s) [upload=335056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 13:15:59.348061896 (381h46m59.219144941s) [upload=335056]: Done uploading segments... Dec 21 13:15:59.348081795 (381h46m59.21916483s) [upload=335056]: Upload finished. err: <nil> Duration: 5.763679637s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..X..X....X........X.....................XXXX...X..X........] duration: 6.137249746s trace: https://jaeger.dev.storj.io/trace/2d9809728bd4cd0c FAIL (2025-12-21 13:22:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: GMCGUAPW66BGV34OH62E5Q66I27LMCCJSGYR5TUWJURFP77KVGJA) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: HMF7YEOHQ5GMCTBMEW3LBCDU2OEUL37IHAHKJTWFID2OUNULAZXQ) context deadline exceeded ======LOGS====== Dec 21 13:15:56.527259459 (381h46m56.398342504s) [upload=335056]: Starting upload Dec 21 13:15:56.527348678 (381h46m56.398431733s) [upload=335056]: Done starting upload Dec 21 13:15:56.527441445 (381h46m56.39852448s) [upload=335056]: Uploading segments... Dec 21 13:15:56.527575821 (381h46m56.398658856s) [upload=335056]: Got next segment. Inline: false Dec 21 13:15:56.528555026 (381h46m56.399638061s) [upload=335056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 13:15:56.528563231 (381h46m56.399646266s) [upload=335056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 13:15:56.775777978 (381h46m56.646861033s) [seg_pos={0 0}, upload=335056]: Begin upload segment... Dec 21 13:15:56.775835256 (381h46m56.646918291s) [seg_pos={0 0}, upload=335056]: Done begin upload segment. Dec 21 13:15:56.775875322 (381h46m56.646958378s) [noise=true , address=145.79.227.43:4157 , piece_id=FUCMIO5FINNCMGYXMDL3G7GY3H7XYQMXSV65F6ILKVWFDIK3BK6Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.775888899 (381h46m56.646971934s) [noise=true , address=45.91.116.14:4189 , piece_id=HMF7YEOHQ5GMCTBMEW3LBCDU2OEUL37IHAHKJTWFID2OUNULAZXQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.776387338 (381h46m56.647470383s) [noise=true , address=145.79.227.48:4301 , piece_id=LSMT6PC7K6BUFGXFFCNLN5TKMB4F7ARL24QRUGLPOO6RP5RNP7YA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.776605343 (381h46m56.647688378s) [noise=true , address=145.79.227.50:4375 , piece_id=U4WW6N3OB7G5T5BZXBYLFWMFERKOEA2VRF7JHRFMNZHKQRVF7U7Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.776799853 (381h46m56.647882898s) [noise=true , address=45.91.116.8:4046 , piece_id=5WFCBJKXGZX63HORQJDVDBDG46BMZ3VE7WWYZPEKXLWUKQ434RYQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777123059 (381h46m56.648206094s) [noise=true , address=145.79.227.46:4603 , piece_id=IBB2HKDX4UOF4OBD6OCPAELN7K5KRKBB3TNCAD4AOAAM3DWT6BDA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777207400 (381h46m56.648290445s) [noise=true , address=145.79.227.52:4445 , piece_id=H7BWJDSSISRYQGOBM5ZKVHR4FKMH5LTONG2T56WTX6IKH4IGIGQQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777533130 (381h46m56.648616165s) [noise=true , address=145.79.227.37:4565 , piece_id=GU35UYW7DUSW2QI7DM4H7GUILILZN2BGPKR5SSPAYC7MTFYI5I4Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777602983 (381h46m56.648686018s) [noise=true , address=45.91.116.13:4156 , piece_id=GMCGUAPW66BGV34OH62E5Q66I27LMCCJSGYR5TUWJURFP77KVGJA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777919996 (381h46m56.649003111s) [noise=true , address=145.79.227.38:4979 , piece_id=A3SZK4HNWGB3AGYKDXVT57CAP5QPEXCJ2OZHIC7XEOEQPNHPV77A, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778138662 (381h46m56.649221697s) [noise=true , address=145.79.227.8:4330 , piece_id=O3BAAICZCQULVD6TSBPOGEHC7FNAB53QNL26OYVX5ED6ESWU64PQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778531932 (381h46m56.649614967s) [noise=true , address=45.91.116.10:4084 , piece_id=7X5ACNCEMTSKIVIDJQ72IPMJGSFC47WO6VN5JANFEMYIUDUZ6ALQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778556999 (381h46m56.649640044s) [noise=true , address=145.79.227.39:4013 , piece_id=GP6DMTAQHDHF2JTUVD6NWPWCS4F3LFMSDGKFK2TUTWQ7U6MXXB5Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778969454 (381h46m56.650052499s) [noise=true , address=77.90.31.10:4519 , piece_id=YWYV2OZRPNYCRFZUXU7I4NPSDZ4TPBPDKZLZTFDP3Z2T6VBD77HA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.779064235 (381h46m56.65014728s) [noise=true , address=145.79.227.29:4943 , piece_id=AR2GVI4J2Q5JY7JE7G7ZYFRNYTW6HHAJEA5AUJTUNEBL6SYEUVUA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.779624251 (381h46m56.650707296s) [noise=true , address=145.79.227.45:4230 , piece_id=WOKT7BUVBTCQMPKTNBWI5IBEVVBCGLJSWVBXPD3DSRXWXS5ZODAA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:58.473831251 (381h46m58.344914296s) [upload=335056]: Next returned nil segment Dec 21 13:15:58.473877700 (381h46m58.344960735s) [upload=335056]: Waiting for error group managing segments... Dec 21 13:15:58.750808791 (381h46m58.621891826s) [noise=true , address=145.79.227.8:4330 , piece_id=O3BAAICZCQULVD6TSBPOGEHC7FNAB53QNL26OYVX5ED6ESWU64PQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.760307128 (381h46m58.631390163s) [noise=true , address=145.79.227.37:4565 , piece_id=GU35UYW7DUSW2QI7DM4H7GUILILZN2BGPKR5SSPAYC7MTFYI5I4Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.761544894 (381h46m58.632627929s) [noise=true , address=145.79.227.50:4375 , piece_id=U4WW6N3OB7G5T5BZXBYLFWMFERKOEA2VRF7JHRFMNZHKQRVF7U7Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.767805456 (381h46m58.638888501s) [noise=true , address=145.79.227.38:4979 , piece_id=A3SZK4HNWGB3AGYKDXVT57CAP5QPEXCJ2OZHIC7XEOEQPNHPV77A, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.770873438 (381h46m58.641956473s) [noise=true , address=145.79.227.48:4301 , piece_id=LSMT6PC7K6BUFGXFFCNLN5TKMB4F7ARL24QRUGLPOO6RP5RNP7YA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.771314767 (381h46m58.642397802s) [noise=true , address=145.79.227.43:4157 , piece_id=FUCMIO5FINNCMGYXMDL3G7GY3H7XYQMXSV65F6ILKVWFDIK3BK6Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.792102381 (381h46m58.663185416s) [noise=true , address=145.79.227.46:4603 , piece_id=IBB2HKDX4UOF4OBD6OCPAELN7K5KRKBB3TNCAD4AOAAM3DWT6BDA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.800571156 (381h46m58.671654191s) [noise=true , address=145.79.227.29:4943 , piece_id=AR2GVI4J2Q5JY7JE7G7ZYFRNYTW6HHAJEA5AUJTUNEBL6SYEUVUA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.827627135 (381h46m58.69871018s) [noise=true , address=145.79.227.52:4445 , piece_id=H7BWJDSSISRYQGOBM5ZKVHR4FKMH5LTONG2T56WTX6IKH4IGIGQQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.832837256 (381h46m58.703920291s) [noise=true , address=145.79.227.45:4230 , piece_id=WOKT7BUVBTCQMPKTNBWI5IBEVVBCGLJSWVBXPD3DSRXWXS5ZODAA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.856094943 (381h46m58.727177988s) [noise=true , address=145.79.227.39:4013 , piece_id=GP6DMTAQHDHF2JTUVD6NWPWCS4F3LFMSDGKFK2TUTWQ7U6MXXB5Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:59.008546104 (381h46m58.879629149s) [noise=true , address=45.91.116.13:4156 , piece_id=GMCGUAPW66BGV34OH62E5Q66I27LMCCJSGYR5TUWJURFP77KVGJA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:59.052840027 (381h46m58.923923062s) [noise=true , address=45.91.116.14:4189 , piece_id=HMF7YEOHQ5GMCTBMEW3LBCDU2OEUL37IHAHKJTWFID2OUNULAZXQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:59.052858953 (381h46m58.923941988s) [seg_pos={0 0}, upload=335056]: Segment reached optimal threshold of 13 pieces. Dec 21 13:15:59.053141110 (381h46m58.924224155s) [noise=true , address=45.91.116.10:4084 , piece_id=7X5ACNCEMTSKIVIDJQ72IPMJGSFC47WO6VN5JANFEMYIUDUZ6ALQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 21 13:15:59.053148975 (381h46m58.92423202s) [noise=true , address=45.91.116.10:4084 , piece_id=7X5ACNCEMTSKIVIDJQ72IPMJGSFC47WO6VN5JANFEMYIUDUZ6ALQ, seg_pos={0 0}, upload=335056]: Long tail cancellation due to optimal threshold Dec 21 13:15:59.053223627 (381h46m58.924306662s) [noise=true , address=45.91.116.8:4046 , piece_id=5WFCBJKXGZX63HORQJDVDBDG46BMZ3VE7WWYZPEKXLWUKQ434RYQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 21 13:15:59.053253414 (381h46m58.924336449s) [noise=true , address=45.91.116.8:4046 , piece_id=5WFCBJKXGZX63HORQJDVDBDG46BMZ3VE7WWYZPEKXLWUKQ434RYQ, seg_pos={0 0}, upload=335056]: Long tail cancellation due to optimal threshold Dec 21 13:15:59.053407067 (381h46m58.924490112s) [noise=true , address=77.90.31.10:4519 , piece_id=YWYV2OZRPNYCRFZUXU7I4NPSDZ4TPBPDKZLZTFDP3Z2T6VBD77HA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 21 13:15:59.053414541 (381h46m58.924497576s) [noise=true , address=77.90.31.10:4519 , piece_id=YWYV2OZRPNYCRFZUXU7I4NPSDZ4TPBPDKZLZTFDP3Z2T6VBD77HA, seg_pos={0 0}, upload=335056]: Long tail cancellation due to optimal threshold Dec 21 13:15:59.053429720 (381h46m58.924512765s) [seg_pos={0 0}, upload=335056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 13:15:59.053500063 (381h46m58.924583109s) [upload=335056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 13:15:59.053507238 (381h46m58.924590283s) [upload=335056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 13:15:59.348061896 (381h46m59.219144941s) [upload=335056]: Done uploading segments... Dec 21 13:15:59.348081795 (381h46m59.21916483s) [upload=335056]: Upload finished. err: <nil> Duration: 6.137280684s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...X..X....X........X.....................XXXX...X..X.......] duration: 5.638682403s trace: https://jaeger.dev.storj.io/trace/2d9809728bd4cd0c FAIL (2025-12-21 13:21:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: GMCGUAPW66BGV34OH62E5Q66I27LMCCJSGYR5TUWJURFP77KVGJA) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: HMF7YEOHQ5GMCTBMEW3LBCDU2OEUL37IHAHKJTWFID2OUNULAZXQ) context deadline exceeded ======LOGS====== Dec 21 13:15:56.527259459 (381h46m56.398342504s) [upload=335056]: Starting upload Dec 21 13:15:56.527348678 (381h46m56.398431733s) [upload=335056]: Done starting upload Dec 21 13:15:56.527441445 (381h46m56.39852448s) [upload=335056]: Uploading segments... Dec 21 13:15:56.527575821 (381h46m56.398658856s) [upload=335056]: Got next segment. Inline: false Dec 21 13:15:56.528555026 (381h46m56.399638061s) [upload=335056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 13:15:56.528563231 (381h46m56.399646266s) [upload=335056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 13:15:56.775777978 (381h46m56.646861033s) [seg_pos={0 0}, upload=335056]: Begin upload segment... Dec 21 13:15:56.775835256 (381h46m56.646918291s) [seg_pos={0 0}, upload=335056]: Done begin upload segment. Dec 21 13:15:56.775875322 (381h46m56.646958378s) [noise=true , address=145.79.227.43:4157 , piece_id=FUCMIO5FINNCMGYXMDL3G7GY3H7XYQMXSV65F6ILKVWFDIK3BK6Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.775888899 (381h46m56.646971934s) [noise=true , address=45.91.116.14:4189 , piece_id=HMF7YEOHQ5GMCTBMEW3LBCDU2OEUL37IHAHKJTWFID2OUNULAZXQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.776387338 (381h46m56.647470383s) [noise=true , address=145.79.227.48:4301 , piece_id=LSMT6PC7K6BUFGXFFCNLN5TKMB4F7ARL24QRUGLPOO6RP5RNP7YA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.776605343 (381h46m56.647688378s) [noise=true , address=145.79.227.50:4375 , piece_id=U4WW6N3OB7G5T5BZXBYLFWMFERKOEA2VRF7JHRFMNZHKQRVF7U7Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.776799853 (381h46m56.647882898s) [noise=true , address=45.91.116.8:4046 , piece_id=5WFCBJKXGZX63HORQJDVDBDG46BMZ3VE7WWYZPEKXLWUKQ434RYQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777123059 (381h46m56.648206094s) [noise=true , address=145.79.227.46:4603 , piece_id=IBB2HKDX4UOF4OBD6OCPAELN7K5KRKBB3TNCAD4AOAAM3DWT6BDA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777207400 (381h46m56.648290445s) [noise=true , address=145.79.227.52:4445 , piece_id=H7BWJDSSISRYQGOBM5ZKVHR4FKMH5LTONG2T56WTX6IKH4IGIGQQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777533130 (381h46m56.648616165s) [noise=true , address=145.79.227.37:4565 , piece_id=GU35UYW7DUSW2QI7DM4H7GUILILZN2BGPKR5SSPAYC7MTFYI5I4Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777602983 (381h46m56.648686018s) [noise=true , address=45.91.116.13:4156 , piece_id=GMCGUAPW66BGV34OH62E5Q66I27LMCCJSGYR5TUWJURFP77KVGJA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777919996 (381h46m56.649003111s) [noise=true , address=145.79.227.38:4979 , piece_id=A3SZK4HNWGB3AGYKDXVT57CAP5QPEXCJ2OZHIC7XEOEQPNHPV77A, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778138662 (381h46m56.649221697s) [noise=true , address=145.79.227.8:4330 , piece_id=O3BAAICZCQULVD6TSBPOGEHC7FNAB53QNL26OYVX5ED6ESWU64PQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778531932 (381h46m56.649614967s) [noise=true , address=45.91.116.10:4084 , piece_id=7X5ACNCEMTSKIVIDJQ72IPMJGSFC47WO6VN5JANFEMYIUDUZ6ALQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778556999 (381h46m56.649640044s) [noise=true , address=145.79.227.39:4013 , piece_id=GP6DMTAQHDHF2JTUVD6NWPWCS4F3LFMSDGKFK2TUTWQ7U6MXXB5Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778969454 (381h46m56.650052499s) [noise=true , address=77.90.31.10:4519 , piece_id=YWYV2OZRPNYCRFZUXU7I4NPSDZ4TPBPDKZLZTFDP3Z2T6VBD77HA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.779064235 (381h46m56.65014728s) [noise=true , address=145.79.227.29:4943 , piece_id=AR2GVI4J2Q5JY7JE7G7ZYFRNYTW6HHAJEA5AUJTUNEBL6SYEUVUA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.779624251 (381h46m56.650707296s) [noise=true , address=145.79.227.45:4230 , piece_id=WOKT7BUVBTCQMPKTNBWI5IBEVVBCGLJSWVBXPD3DSRXWXS5ZODAA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:58.473831251 (381h46m58.344914296s) [upload=335056]: Next returned nil segment Dec 21 13:15:58.473877700 (381h46m58.344960735s) [upload=335056]: Waiting for error group managing segments... Dec 21 13:15:58.750808791 (381h46m58.621891826s) [noise=true , address=145.79.227.8:4330 , piece_id=O3BAAICZCQULVD6TSBPOGEHC7FNAB53QNL26OYVX5ED6ESWU64PQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.760307128 (381h46m58.631390163s) [noise=true , address=145.79.227.37:4565 , piece_id=GU35UYW7DUSW2QI7DM4H7GUILILZN2BGPKR5SSPAYC7MTFYI5I4Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.761544894 (381h46m58.632627929s) [noise=true , address=145.79.227.50:4375 , piece_id=U4WW6N3OB7G5T5BZXBYLFWMFERKOEA2VRF7JHRFMNZHKQRVF7U7Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.767805456 (381h46m58.638888501s) [noise=true , address=145.79.227.38:4979 , piece_id=A3SZK4HNWGB3AGYKDXVT57CAP5QPEXCJ2OZHIC7XEOEQPNHPV77A, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.770873438 (381h46m58.641956473s) [noise=true , address=145.79.227.48:4301 , piece_id=LSMT6PC7K6BUFGXFFCNLN5TKMB4F7ARL24QRUGLPOO6RP5RNP7YA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.771314767 (381h46m58.642397802s) [noise=true , address=145.79.227.43:4157 , piece_id=FUCMIO5FINNCMGYXMDL3G7GY3H7XYQMXSV65F6ILKVWFDIK3BK6Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.792102381 (381h46m58.663185416s) [noise=true , address=145.79.227.46:4603 , piece_id=IBB2HKDX4UOF4OBD6OCPAELN7K5KRKBB3TNCAD4AOAAM3DWT6BDA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.800571156 (381h46m58.671654191s) [noise=true , address=145.79.227.29:4943 , piece_id=AR2GVI4J2Q5JY7JE7G7ZYFRNYTW6HHAJEA5AUJTUNEBL6SYEUVUA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.827627135 (381h46m58.69871018s) [noise=true , address=145.79.227.52:4445 , piece_id=H7BWJDSSISRYQGOBM5ZKVHR4FKMH5LTONG2T56WTX6IKH4IGIGQQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.832837256 (381h46m58.703920291s) [noise=true , address=145.79.227.45:4230 , piece_id=WOKT7BUVBTCQMPKTNBWI5IBEVVBCGLJSWVBXPD3DSRXWXS5ZODAA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.856094943 (381h46m58.727177988s) [noise=true , address=145.79.227.39:4013 , piece_id=GP6DMTAQHDHF2JTUVD6NWPWCS4F3LFMSDGKFK2TUTWQ7U6MXXB5Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:59.008546104 (381h46m58.879629149s) [noise=true , address=45.91.116.13:4156 , piece_id=GMCGUAPW66BGV34OH62E5Q66I27LMCCJSGYR5TUWJURFP77KVGJA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:59.052840027 (381h46m58.923923062s) [noise=true , address=45.91.116.14:4189 , piece_id=HMF7YEOHQ5GMCTBMEW3LBCDU2OEUL37IHAHKJTWFID2OUNULAZXQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:59.052858953 (381h46m58.923941988s) [seg_pos={0 0}, upload=335056]: Segment reached optimal threshold of 13 pieces. Dec 21 13:15:59.053141110 (381h46m58.924224155s) [noise=true , address=45.91.116.10:4084 , piece_id=7X5ACNCEMTSKIVIDJQ72IPMJGSFC47WO6VN5JANFEMYIUDUZ6ALQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 21 13:15:59.053148975 (381h46m58.92423202s) [noise=true , address=45.91.116.10:4084 , piece_id=7X5ACNCEMTSKIVIDJQ72IPMJGSFC47WO6VN5JANFEMYIUDUZ6ALQ, seg_pos={0 0}, upload=335056]: Long tail cancellation due to optimal threshold Dec 21 13:15:59.053223627 (381h46m58.924306662s) [noise=true , address=45.91.116.8:4046 , piece_id=5WFCBJKXGZX63HORQJDVDBDG46BMZ3VE7WWYZPEKXLWUKQ434RYQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 21 13:15:59.053253414 (381h46m58.924336449s) [noise=true , address=45.91.116.8:4046 , piece_id=5WFCBJKXGZX63HORQJDVDBDG46BMZ3VE7WWYZPEKXLWUKQ434RYQ, seg_pos={0 0}, upload=335056]: Long tail cancellation due to optimal threshold Dec 21 13:15:59.053407067 (381h46m58.924490112s) [noise=true , address=77.90.31.10:4519 , piece_id=YWYV2OZRPNYCRFZUXU7I4NPSDZ4TPBPDKZLZTFDP3Z2T6VBD77HA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 21 13:15:59.053414541 (381h46m58.924497576s) [noise=true , address=77.90.31.10:4519 , piece_id=YWYV2OZRPNYCRFZUXU7I4NPSDZ4TPBPDKZLZTFDP3Z2T6VBD77HA, seg_pos={0 0}, upload=335056]: Long tail cancellation due to optimal threshold Dec 21 13:15:59.053429720 (381h46m58.924512765s) [seg_pos={0 0}, upload=335056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 13:15:59.053500063 (381h46m58.924583109s) [upload=335056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 13:15:59.053507238 (381h46m58.924590283s) [upload=335056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 13:15:59.348061896 (381h46m59.219144941s) [upload=335056]: Done uploading segments... Dec 21 13:15:59.348081795 (381h46m59.21916483s) [upload=335056]: Upload finished. err: <nil> Duration: 5.638713461s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [....X..X....X........X.....................XXXX...X..X......] duration: 5.745361078s trace: https://jaeger.dev.storj.io/trace/2d9809728bd4cd0c FAIL (2025-12-21 13:20:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: GMCGUAPW66BGV34OH62E5Q66I27LMCCJSGYR5TUWJURFP77KVGJA) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: HMF7YEOHQ5GMCTBMEW3LBCDU2OEUL37IHAHKJTWFID2OUNULAZXQ) context deadline exceeded ======LOGS====== Dec 21 13:15:56.527259459 (381h46m56.398342504s) [upload=335056]: Starting upload Dec 21 13:15:56.527348678 (381h46m56.398431733s) [upload=335056]: Done starting upload Dec 21 13:15:56.527441445 (381h46m56.39852448s) [upload=335056]: Uploading segments... Dec 21 13:15:56.527575821 (381h46m56.398658856s) [upload=335056]: Got next segment. Inline: false Dec 21 13:15:56.528555026 (381h46m56.399638061s) [upload=335056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 13:15:56.528563231 (381h46m56.399646266s) [upload=335056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 13:15:56.775777978 (381h46m56.646861033s) [seg_pos={0 0}, upload=335056]: Begin upload segment... Dec 21 13:15:56.775835256 (381h46m56.646918291s) [seg_pos={0 0}, upload=335056]: Done begin upload segment. Dec 21 13:15:56.775875322 (381h46m56.646958378s) [noise=true , address=145.79.227.43:4157 , piece_id=FUCMIO5FINNCMGYXMDL3G7GY3H7XYQMXSV65F6ILKVWFDIK3BK6Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.775888899 (381h46m56.646971934s) [noise=true , address=45.91.116.14:4189 , piece_id=HMF7YEOHQ5GMCTBMEW3LBCDU2OEUL37IHAHKJTWFID2OUNULAZXQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.776387338 (381h46m56.647470383s) [noise=true , address=145.79.227.48:4301 , piece_id=LSMT6PC7K6BUFGXFFCNLN5TKMB4F7ARL24QRUGLPOO6RP5RNP7YA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.776605343 (381h46m56.647688378s) [noise=true , address=145.79.227.50:4375 , piece_id=U4WW6N3OB7G5T5BZXBYLFWMFERKOEA2VRF7JHRFMNZHKQRVF7U7Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.776799853 (381h46m56.647882898s) [noise=true , address=45.91.116.8:4046 , piece_id=5WFCBJKXGZX63HORQJDVDBDG46BMZ3VE7WWYZPEKXLWUKQ434RYQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777123059 (381h46m56.648206094s) [noise=true , address=145.79.227.46:4603 , piece_id=IBB2HKDX4UOF4OBD6OCPAELN7K5KRKBB3TNCAD4AOAAM3DWT6BDA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777207400 (381h46m56.648290445s) [noise=true , address=145.79.227.52:4445 , piece_id=H7BWJDSSISRYQGOBM5ZKVHR4FKMH5LTONG2T56WTX6IKH4IGIGQQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777533130 (381h46m56.648616165s) [noise=true , address=145.79.227.37:4565 , piece_id=GU35UYW7DUSW2QI7DM4H7GUILILZN2BGPKR5SSPAYC7MTFYI5I4Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777602983 (381h46m56.648686018s) [noise=true , address=45.91.116.13:4156 , piece_id=GMCGUAPW66BGV34OH62E5Q66I27LMCCJSGYR5TUWJURFP77KVGJA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777919996 (381h46m56.649003111s) [noise=true , address=145.79.227.38:4979 , piece_id=A3SZK4HNWGB3AGYKDXVT57CAP5QPEXCJ2OZHIC7XEOEQPNHPV77A, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778138662 (381h46m56.649221697s) [noise=true , address=145.79.227.8:4330 , piece_id=O3BAAICZCQULVD6TSBPOGEHC7FNAB53QNL26OYVX5ED6ESWU64PQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778531932 (381h46m56.649614967s) [noise=true , address=45.91.116.10:4084 , piece_id=7X5ACNCEMTSKIVIDJQ72IPMJGSFC47WO6VN5JANFEMYIUDUZ6ALQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778556999 (381h46m56.649640044s) [noise=true , address=145.79.227.39:4013 , piece_id=GP6DMTAQHDHF2JTUVD6NWPWCS4F3LFMSDGKFK2TUTWQ7U6MXXB5Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778969454 (381h46m56.650052499s) [noise=true , address=77.90.31.10:4519 , piece_id=YWYV2OZRPNYCRFZUXU7I4NPSDZ4TPBPDKZLZTFDP3Z2T6VBD77HA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.779064235 (381h46m56.65014728s) [noise=true , address=145.79.227.29:4943 , piece_id=AR2GVI4J2Q5JY7JE7G7ZYFRNYTW6HHAJEA5AUJTUNEBL6SYEUVUA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.779624251 (381h46m56.650707296s) [noise=true , address=145.79.227.45:4230 , piece_id=WOKT7BUVBTCQMPKTNBWI5IBEVVBCGLJSWVBXPD3DSRXWXS5ZODAA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:58.473831251 (381h46m58.344914296s) [upload=335056]: Next returned nil segment Dec 21 13:15:58.473877700 (381h46m58.344960735s) [upload=335056]: Waiting for error group managing segments... Dec 21 13:15:58.750808791 (381h46m58.621891826s) [noise=true , address=145.79.227.8:4330 , piece_id=O3BAAICZCQULVD6TSBPOGEHC7FNAB53QNL26OYVX5ED6ESWU64PQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.760307128 (381h46m58.631390163s) [noise=true , address=145.79.227.37:4565 , piece_id=GU35UYW7DUSW2QI7DM4H7GUILILZN2BGPKR5SSPAYC7MTFYI5I4Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.761544894 (381h46m58.632627929s) [noise=true , address=145.79.227.50:4375 , piece_id=U4WW6N3OB7G5T5BZXBYLFWMFERKOEA2VRF7JHRFMNZHKQRVF7U7Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.767805456 (381h46m58.638888501s) [noise=true , address=145.79.227.38:4979 , piece_id=A3SZK4HNWGB3AGYKDXVT57CAP5QPEXCJ2OZHIC7XEOEQPNHPV77A, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.770873438 (381h46m58.641956473s) [noise=true , address=145.79.227.48:4301 , piece_id=LSMT6PC7K6BUFGXFFCNLN5TKMB4F7ARL24QRUGLPOO6RP5RNP7YA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.771314767 (381h46m58.642397802s) [noise=true , address=145.79.227.43:4157 , piece_id=FUCMIO5FINNCMGYXMDL3G7GY3H7XYQMXSV65F6ILKVWFDIK3BK6Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.792102381 (381h46m58.663185416s) [noise=true , address=145.79.227.46:4603 , piece_id=IBB2HKDX4UOF4OBD6OCPAELN7K5KRKBB3TNCAD4AOAAM3DWT6BDA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.800571156 (381h46m58.671654191s) [noise=true , address=145.79.227.29:4943 , piece_id=AR2GVI4J2Q5JY7JE7G7ZYFRNYTW6HHAJEA5AUJTUNEBL6SYEUVUA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.827627135 (381h46m58.69871018s) [noise=true , address=145.79.227.52:4445 , piece_id=H7BWJDSSISRYQGOBM5ZKVHR4FKMH5LTONG2T56WTX6IKH4IGIGQQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.832837256 (381h46m58.703920291s) [noise=true , address=145.79.227.45:4230 , piece_id=WOKT7BUVBTCQMPKTNBWI5IBEVVBCGLJSWVBXPD3DSRXWXS5ZODAA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.856094943 (381h46m58.727177988s) [noise=true , address=145.79.227.39:4013 , piece_id=GP6DMTAQHDHF2JTUVD6NWPWCS4F3LFMSDGKFK2TUTWQ7U6MXXB5Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:59.008546104 (381h46m58.879629149s) [noise=true , address=45.91.116.13:4156 , piece_id=GMCGUAPW66BGV34OH62E5Q66I27LMCCJSGYR5TUWJURFP77KVGJA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:59.052840027 (381h46m58.923923062s) [noise=true , address=45.91.116.14:4189 , piece_id=HMF7YEOHQ5GMCTBMEW3LBCDU2OEUL37IHAHKJTWFID2OUNULAZXQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:59.052858953 (381h46m58.923941988s) [seg_pos={0 0}, upload=335056]: Segment reached optimal threshold of 13 pieces. Dec 21 13:15:59.053141110 (381h46m58.924224155s) [noise=true , address=45.91.116.10:4084 , piece_id=7X5ACNCEMTSKIVIDJQ72IPMJGSFC47WO6VN5JANFEMYIUDUZ6ALQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 21 13:15:59.053148975 (381h46m58.92423202s) [noise=true , address=45.91.116.10:4084 , piece_id=7X5ACNCEMTSKIVIDJQ72IPMJGSFC47WO6VN5JANFEMYIUDUZ6ALQ, seg_pos={0 0}, upload=335056]: Long tail cancellation due to optimal threshold Dec 21 13:15:59.053223627 (381h46m58.924306662s) [noise=true , address=45.91.116.8:4046 , piece_id=5WFCBJKXGZX63HORQJDVDBDG46BMZ3VE7WWYZPEKXLWUKQ434RYQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 21 13:15:59.053253414 (381h46m58.924336449s) [noise=true , address=45.91.116.8:4046 , piece_id=5WFCBJKXGZX63HORQJDVDBDG46BMZ3VE7WWYZPEKXLWUKQ434RYQ, seg_pos={0 0}, upload=335056]: Long tail cancellation due to optimal threshold Dec 21 13:15:59.053407067 (381h46m58.924490112s) [noise=true , address=77.90.31.10:4519 , piece_id=YWYV2OZRPNYCRFZUXU7I4NPSDZ4TPBPDKZLZTFDP3Z2T6VBD77HA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 21 13:15:59.053414541 (381h46m58.924497576s) [noise=true , address=77.90.31.10:4519 , piece_id=YWYV2OZRPNYCRFZUXU7I4NPSDZ4TPBPDKZLZTFDP3Z2T6VBD77HA, seg_pos={0 0}, upload=335056]: Long tail cancellation due to optimal threshold Dec 21 13:15:59.053429720 (381h46m58.924512765s) [seg_pos={0 0}, upload=335056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 13:15:59.053500063 (381h46m58.924583109s) [upload=335056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 13:15:59.053507238 (381h46m58.924590283s) [upload=335056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 13:15:59.348061896 (381h46m59.219144941s) [upload=335056]: Done uploading segments... Dec 21 13:15:59.348081795 (381h46m59.21916483s) [upload=335056]: Upload finished. err: <nil> Duration: 5.745384052s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.....X..X....X........X.....................XXXX...X..X.....] duration: 5.712957184s trace: https://jaeger.dev.storj.io/trace/2d9809728bd4cd0c FAIL (2025-12-21 13:20:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: GMCGUAPW66BGV34OH62E5Q66I27LMCCJSGYR5TUWJURFP77KVGJA) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: HMF7YEOHQ5GMCTBMEW3LBCDU2OEUL37IHAHKJTWFID2OUNULAZXQ) context deadline exceeded ======LOGS====== Dec 21 13:15:56.527259459 (381h46m56.398342504s) [upload=335056]: Starting upload Dec 21 13:15:56.527348678 (381h46m56.398431733s) [upload=335056]: Done starting upload Dec 21 13:15:56.527441445 (381h46m56.39852448s) [upload=335056]: Uploading segments... Dec 21 13:15:56.527575821 (381h46m56.398658856s) [upload=335056]: Got next segment. Inline: false Dec 21 13:15:56.528555026 (381h46m56.399638061s) [upload=335056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 13:15:56.528563231 (381h46m56.399646266s) [upload=335056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 13:15:56.775777978 (381h46m56.646861033s) [seg_pos={0 0}, upload=335056]: Begin upload segment... Dec 21 13:15:56.775835256 (381h46m56.646918291s) [seg_pos={0 0}, upload=335056]: Done begin upload segment. Dec 21 13:15:56.775875322 (381h46m56.646958378s) [noise=true , address=145.79.227.43:4157 , piece_id=FUCMIO5FINNCMGYXMDL3G7GY3H7XYQMXSV65F6ILKVWFDIK3BK6Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.775888899 (381h46m56.646971934s) [noise=true , address=45.91.116.14:4189 , piece_id=HMF7YEOHQ5GMCTBMEW3LBCDU2OEUL37IHAHKJTWFID2OUNULAZXQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.776387338 (381h46m56.647470383s) [noise=true , address=145.79.227.48:4301 , piece_id=LSMT6PC7K6BUFGXFFCNLN5TKMB4F7ARL24QRUGLPOO6RP5RNP7YA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.776605343 (381h46m56.647688378s) [noise=true , address=145.79.227.50:4375 , piece_id=U4WW6N3OB7G5T5BZXBYLFWMFERKOEA2VRF7JHRFMNZHKQRVF7U7Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.776799853 (381h46m56.647882898s) [noise=true , address=45.91.116.8:4046 , piece_id=5WFCBJKXGZX63HORQJDVDBDG46BMZ3VE7WWYZPEKXLWUKQ434RYQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777123059 (381h46m56.648206094s) [noise=true , address=145.79.227.46:4603 , piece_id=IBB2HKDX4UOF4OBD6OCPAELN7K5KRKBB3TNCAD4AOAAM3DWT6BDA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777207400 (381h46m56.648290445s) [noise=true , address=145.79.227.52:4445 , piece_id=H7BWJDSSISRYQGOBM5ZKVHR4FKMH5LTONG2T56WTX6IKH4IGIGQQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777533130 (381h46m56.648616165s) [noise=true , address=145.79.227.37:4565 , piece_id=GU35UYW7DUSW2QI7DM4H7GUILILZN2BGPKR5SSPAYC7MTFYI5I4Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777602983 (381h46m56.648686018s) [noise=true , address=45.91.116.13:4156 , piece_id=GMCGUAPW66BGV34OH62E5Q66I27LMCCJSGYR5TUWJURFP77KVGJA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777919996 (381h46m56.649003111s) [noise=true , address=145.79.227.38:4979 , piece_id=A3SZK4HNWGB3AGYKDXVT57CAP5QPEXCJ2OZHIC7XEOEQPNHPV77A, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778138662 (381h46m56.649221697s) [noise=true , address=145.79.227.8:4330 , piece_id=O3BAAICZCQULVD6TSBPOGEHC7FNAB53QNL26OYVX5ED6ESWU64PQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778531932 (381h46m56.649614967s) [noise=true , address=45.91.116.10:4084 , piece_id=7X5ACNCEMTSKIVIDJQ72IPMJGSFC47WO6VN5JANFEMYIUDUZ6ALQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778556999 (381h46m56.649640044s) [noise=true , address=145.79.227.39:4013 , piece_id=GP6DMTAQHDHF2JTUVD6NWPWCS4F3LFMSDGKFK2TUTWQ7U6MXXB5Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778969454 (381h46m56.650052499s) [noise=true , address=77.90.31.10:4519 , piece_id=YWYV2OZRPNYCRFZUXU7I4NPSDZ4TPBPDKZLZTFDP3Z2T6VBD77HA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.779064235 (381h46m56.65014728s) [noise=true , address=145.79.227.29:4943 , piece_id=AR2GVI4J2Q5JY7JE7G7ZYFRNYTW6HHAJEA5AUJTUNEBL6SYEUVUA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.779624251 (381h46m56.650707296s) [noise=true , address=145.79.227.45:4230 , piece_id=WOKT7BUVBTCQMPKTNBWI5IBEVVBCGLJSWVBXPD3DSRXWXS5ZODAA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:58.473831251 (381h46m58.344914296s) [upload=335056]: Next returned nil segment Dec 21 13:15:58.473877700 (381h46m58.344960735s) [upload=335056]: Waiting for error group managing segments... Dec 21 13:15:58.750808791 (381h46m58.621891826s) [noise=true , address=145.79.227.8:4330 , piece_id=O3BAAICZCQULVD6TSBPOGEHC7FNAB53QNL26OYVX5ED6ESWU64PQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.760307128 (381h46m58.631390163s) [noise=true , address=145.79.227.37:4565 , piece_id=GU35UYW7DUSW2QI7DM4H7GUILILZN2BGPKR5SSPAYC7MTFYI5I4Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.761544894 (381h46m58.632627929s) [noise=true , address=145.79.227.50:4375 , piece_id=U4WW6N3OB7G5T5BZXBYLFWMFERKOEA2VRF7JHRFMNZHKQRVF7U7Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.767805456 (381h46m58.638888501s) [noise=true , address=145.79.227.38:4979 , piece_id=A3SZK4HNWGB3AGYKDXVT57CAP5QPEXCJ2OZHIC7XEOEQPNHPV77A, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.770873438 (381h46m58.641956473s) [noise=true , address=145.79.227.48:4301 , piece_id=LSMT6PC7K6BUFGXFFCNLN5TKMB4F7ARL24QRUGLPOO6RP5RNP7YA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.771314767 (381h46m58.642397802s) [noise=true , address=145.79.227.43:4157 , piece_id=FUCMIO5FINNCMGYXMDL3G7GY3H7XYQMXSV65F6ILKVWFDIK3BK6Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.792102381 (381h46m58.663185416s) [noise=true , address=145.79.227.46:4603 , piece_id=IBB2HKDX4UOF4OBD6OCPAELN7K5KRKBB3TNCAD4AOAAM3DWT6BDA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.800571156 (381h46m58.671654191s) [noise=true , address=145.79.227.29:4943 , piece_id=AR2GVI4J2Q5JY7JE7G7ZYFRNYTW6HHAJEA5AUJTUNEBL6SYEUVUA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.827627135 (381h46m58.69871018s) [noise=true , address=145.79.227.52:4445 , piece_id=H7BWJDSSISRYQGOBM5ZKVHR4FKMH5LTONG2T56WTX6IKH4IGIGQQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.832837256 (381h46m58.703920291s) [noise=true , address=145.79.227.45:4230 , piece_id=WOKT7BUVBTCQMPKTNBWI5IBEVVBCGLJSWVBXPD3DSRXWXS5ZODAA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.856094943 (381h46m58.727177988s) [noise=true , address=145.79.227.39:4013 , piece_id=GP6DMTAQHDHF2JTUVD6NWPWCS4F3LFMSDGKFK2TUTWQ7U6MXXB5Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:59.008546104 (381h46m58.879629149s) [noise=true , address=45.91.116.13:4156 , piece_id=GMCGUAPW66BGV34OH62E5Q66I27LMCCJSGYR5TUWJURFP77KVGJA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:59.052840027 (381h46m58.923923062s) [noise=true , address=45.91.116.14:4189 , piece_id=HMF7YEOHQ5GMCTBMEW3LBCDU2OEUL37IHAHKJTWFID2OUNULAZXQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:59.052858953 (381h46m58.923941988s) [seg_pos={0 0}, upload=335056]: Segment reached optimal threshold of 13 pieces. Dec 21 13:15:59.053141110 (381h46m58.924224155s) [noise=true , address=45.91.116.10:4084 , piece_id=7X5ACNCEMTSKIVIDJQ72IPMJGSFC47WO6VN5JANFEMYIUDUZ6ALQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 21 13:15:59.053148975 (381h46m58.92423202s) [noise=true , address=45.91.116.10:4084 , piece_id=7X5ACNCEMTSKIVIDJQ72IPMJGSFC47WO6VN5JANFEMYIUDUZ6ALQ, seg_pos={0 0}, upload=335056]: Long tail cancellation due to optimal threshold Dec 21 13:15:59.053223627 (381h46m58.924306662s) [noise=true , address=45.91.116.8:4046 , piece_id=5WFCBJKXGZX63HORQJDVDBDG46BMZ3VE7WWYZPEKXLWUKQ434RYQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 21 13:15:59.053253414 (381h46m58.924336449s) [noise=true , address=45.91.116.8:4046 , piece_id=5WFCBJKXGZX63HORQJDVDBDG46BMZ3VE7WWYZPEKXLWUKQ434RYQ, seg_pos={0 0}, upload=335056]: Long tail cancellation due to optimal threshold Dec 21 13:15:59.053407067 (381h46m58.924490112s) [noise=true , address=77.90.31.10:4519 , piece_id=YWYV2OZRPNYCRFZUXU7I4NPSDZ4TPBPDKZLZTFDP3Z2T6VBD77HA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 21 13:15:59.053414541 (381h46m58.924497576s) [noise=true , address=77.90.31.10:4519 , piece_id=YWYV2OZRPNYCRFZUXU7I4NPSDZ4TPBPDKZLZTFDP3Z2T6VBD77HA, seg_pos={0 0}, upload=335056]: Long tail cancellation due to optimal threshold Dec 21 13:15:59.053429720 (381h46m58.924512765s) [seg_pos={0 0}, upload=335056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 13:15:59.053500063 (381h46m58.924583109s) [upload=335056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 13:15:59.053507238 (381h46m58.924590283s) [upload=335056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 13:15:59.348061896 (381h46m59.219144941s) [upload=335056]: Done uploading segments... Dec 21 13:15:59.348081795 (381h46m59.21916483s) [upload=335056]: Upload finished. err: <nil> Duration: 5.712985949s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X.....X..X....X........X.....................XXXX...X..X....] duration: 5.838303335s trace: https://jaeger.dev.storj.io/trace/2d9809728bd4cd0c FAIL (2025-12-21 13:19:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: GMCGUAPW66BGV34OH62E5Q66I27LMCCJSGYR5TUWJURFP77KVGJA) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: HMF7YEOHQ5GMCTBMEW3LBCDU2OEUL37IHAHKJTWFID2OUNULAZXQ) context deadline exceeded ======LOGS====== Dec 21 13:15:56.527259459 (381h46m56.398342504s) [upload=335056]: Starting upload Dec 21 13:15:56.527348678 (381h46m56.398431733s) [upload=335056]: Done starting upload Dec 21 13:15:56.527441445 (381h46m56.39852448s) [upload=335056]: Uploading segments... Dec 21 13:15:56.527575821 (381h46m56.398658856s) [upload=335056]: Got next segment. Inline: false Dec 21 13:15:56.528555026 (381h46m56.399638061s) [upload=335056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 13:15:56.528563231 (381h46m56.399646266s) [upload=335056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 13:15:56.775777978 (381h46m56.646861033s) [seg_pos={0 0}, upload=335056]: Begin upload segment... Dec 21 13:15:56.775835256 (381h46m56.646918291s) [seg_pos={0 0}, upload=335056]: Done begin upload segment. Dec 21 13:15:56.775875322 (381h46m56.646958378s) [noise=true , address=145.79.227.43:4157 , piece_id=FUCMIO5FINNCMGYXMDL3G7GY3H7XYQMXSV65F6ILKVWFDIK3BK6Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.775888899 (381h46m56.646971934s) [noise=true , address=45.91.116.14:4189 , piece_id=HMF7YEOHQ5GMCTBMEW3LBCDU2OEUL37IHAHKJTWFID2OUNULAZXQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.776387338 (381h46m56.647470383s) [noise=true , address=145.79.227.48:4301 , piece_id=LSMT6PC7K6BUFGXFFCNLN5TKMB4F7ARL24QRUGLPOO6RP5RNP7YA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.776605343 (381h46m56.647688378s) [noise=true , address=145.79.227.50:4375 , piece_id=U4WW6N3OB7G5T5BZXBYLFWMFERKOEA2VRF7JHRFMNZHKQRVF7U7Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.776799853 (381h46m56.647882898s) [noise=true , address=45.91.116.8:4046 , piece_id=5WFCBJKXGZX63HORQJDVDBDG46BMZ3VE7WWYZPEKXLWUKQ434RYQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777123059 (381h46m56.648206094s) [noise=true , address=145.79.227.46:4603 , piece_id=IBB2HKDX4UOF4OBD6OCPAELN7K5KRKBB3TNCAD4AOAAM3DWT6BDA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777207400 (381h46m56.648290445s) [noise=true , address=145.79.227.52:4445 , piece_id=H7BWJDSSISRYQGOBM5ZKVHR4FKMH5LTONG2T56WTX6IKH4IGIGQQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777533130 (381h46m56.648616165s) [noise=true , address=145.79.227.37:4565 , piece_id=GU35UYW7DUSW2QI7DM4H7GUILILZN2BGPKR5SSPAYC7MTFYI5I4Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777602983 (381h46m56.648686018s) [noise=true , address=45.91.116.13:4156 , piece_id=GMCGUAPW66BGV34OH62E5Q66I27LMCCJSGYR5TUWJURFP77KVGJA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777919996 (381h46m56.649003111s) [noise=true , address=145.79.227.38:4979 , piece_id=A3SZK4HNWGB3AGYKDXVT57CAP5QPEXCJ2OZHIC7XEOEQPNHPV77A, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778138662 (381h46m56.649221697s) [noise=true , address=145.79.227.8:4330 , piece_id=O3BAAICZCQULVD6TSBPOGEHC7FNAB53QNL26OYVX5ED6ESWU64PQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778531932 (381h46m56.649614967s) [noise=true , address=45.91.116.10:4084 , piece_id=7X5ACNCEMTSKIVIDJQ72IPMJGSFC47WO6VN5JANFEMYIUDUZ6ALQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778556999 (381h46m56.649640044s) [noise=true , address=145.79.227.39:4013 , piece_id=GP6DMTAQHDHF2JTUVD6NWPWCS4F3LFMSDGKFK2TUTWQ7U6MXXB5Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778969454 (381h46m56.650052499s) [noise=true , address=77.90.31.10:4519 , piece_id=YWYV2OZRPNYCRFZUXU7I4NPSDZ4TPBPDKZLZTFDP3Z2T6VBD77HA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.779064235 (381h46m56.65014728s) [noise=true , address=145.79.227.29:4943 , piece_id=AR2GVI4J2Q5JY7JE7G7ZYFRNYTW6HHAJEA5AUJTUNEBL6SYEUVUA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.779624251 (381h46m56.650707296s) [noise=true , address=145.79.227.45:4230 , piece_id=WOKT7BUVBTCQMPKTNBWI5IBEVVBCGLJSWVBXPD3DSRXWXS5ZODAA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:58.473831251 (381h46m58.344914296s) [upload=335056]: Next returned nil segment Dec 21 13:15:58.473877700 (381h46m58.344960735s) [upload=335056]: Waiting for error group managing segments... Dec 21 13:15:58.750808791 (381h46m58.621891826s) [noise=true , address=145.79.227.8:4330 , piece_id=O3BAAICZCQULVD6TSBPOGEHC7FNAB53QNL26OYVX5ED6ESWU64PQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.760307128 (381h46m58.631390163s) [noise=true , address=145.79.227.37:4565 , piece_id=GU35UYW7DUSW2QI7DM4H7GUILILZN2BGPKR5SSPAYC7MTFYI5I4Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.761544894 (381h46m58.632627929s) [noise=true , address=145.79.227.50:4375 , piece_id=U4WW6N3OB7G5T5BZXBYLFWMFERKOEA2VRF7JHRFMNZHKQRVF7U7Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.767805456 (381h46m58.638888501s) [noise=true , address=145.79.227.38:4979 , piece_id=A3SZK4HNWGB3AGYKDXVT57CAP5QPEXCJ2OZHIC7XEOEQPNHPV77A, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.770873438 (381h46m58.641956473s) [noise=true , address=145.79.227.48:4301 , piece_id=LSMT6PC7K6BUFGXFFCNLN5TKMB4F7ARL24QRUGLPOO6RP5RNP7YA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.771314767 (381h46m58.642397802s) [noise=true , address=145.79.227.43:4157 , piece_id=FUCMIO5FINNCMGYXMDL3G7GY3H7XYQMXSV65F6ILKVWFDIK3BK6Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.792102381 (381h46m58.663185416s) [noise=true , address=145.79.227.46:4603 , piece_id=IBB2HKDX4UOF4OBD6OCPAELN7K5KRKBB3TNCAD4AOAAM3DWT6BDA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.800571156 (381h46m58.671654191s) [noise=true , address=145.79.227.29:4943 , piece_id=AR2GVI4J2Q5JY7JE7G7ZYFRNYTW6HHAJEA5AUJTUNEBL6SYEUVUA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.827627135 (381h46m58.69871018s) [noise=true , address=145.79.227.52:4445 , piece_id=H7BWJDSSISRYQGOBM5ZKVHR4FKMH5LTONG2T56WTX6IKH4IGIGQQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.832837256 (381h46m58.703920291s) [noise=true , address=145.79.227.45:4230 , piece_id=WOKT7BUVBTCQMPKTNBWI5IBEVVBCGLJSWVBXPD3DSRXWXS5ZODAA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.856094943 (381h46m58.727177988s) [noise=true , address=145.79.227.39:4013 , piece_id=GP6DMTAQHDHF2JTUVD6NWPWCS4F3LFMSDGKFK2TUTWQ7U6MXXB5Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:59.008546104 (381h46m58.879629149s) [noise=true , address=45.91.116.13:4156 , piece_id=GMCGUAPW66BGV34OH62E5Q66I27LMCCJSGYR5TUWJURFP77KVGJA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:59.052840027 (381h46m58.923923062s) [noise=true , address=45.91.116.14:4189 , piece_id=HMF7YEOHQ5GMCTBMEW3LBCDU2OEUL37IHAHKJTWFID2OUNULAZXQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:59.052858953 (381h46m58.923941988s) [seg_pos={0 0}, upload=335056]: Segment reached optimal threshold of 13 pieces. Dec 21 13:15:59.053141110 (381h46m58.924224155s) [noise=true , address=45.91.116.10:4084 , piece_id=7X5ACNCEMTSKIVIDJQ72IPMJGSFC47WO6VN5JANFEMYIUDUZ6ALQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 21 13:15:59.053148975 (381h46m58.92423202s) [noise=true , address=45.91.116.10:4084 , piece_id=7X5ACNCEMTSKIVIDJQ72IPMJGSFC47WO6VN5JANFEMYIUDUZ6ALQ, seg_pos={0 0}, upload=335056]: Long tail cancellation due to optimal threshold Dec 21 13:15:59.053223627 (381h46m58.924306662s) [noise=true , address=45.91.116.8:4046 , piece_id=5WFCBJKXGZX63HORQJDVDBDG46BMZ3VE7WWYZPEKXLWUKQ434RYQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 21 13:15:59.053253414 (381h46m58.924336449s) [noise=true , address=45.91.116.8:4046 , piece_id=5WFCBJKXGZX63HORQJDVDBDG46BMZ3VE7WWYZPEKXLWUKQ434RYQ, seg_pos={0 0}, upload=335056]: Long tail cancellation due to optimal threshold Dec 21 13:15:59.053407067 (381h46m58.924490112s) [noise=true , address=77.90.31.10:4519 , piece_id=YWYV2OZRPNYCRFZUXU7I4NPSDZ4TPBPDKZLZTFDP3Z2T6VBD77HA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 21 13:15:59.053414541 (381h46m58.924497576s) [noise=true , address=77.90.31.10:4519 , piece_id=YWYV2OZRPNYCRFZUXU7I4NPSDZ4TPBPDKZLZTFDP3Z2T6VBD77HA, seg_pos={0 0}, upload=335056]: Long tail cancellation due to optimal threshold Dec 21 13:15:59.053429720 (381h46m58.924512765s) [seg_pos={0 0}, upload=335056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 13:15:59.053500063 (381h46m58.924583109s) [upload=335056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 13:15:59.053507238 (381h46m58.924590283s) [upload=335056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 13:15:59.348061896 (381h46m59.219144941s) [upload=335056]: Done uploading segments... Dec 21 13:15:59.348081795 (381h46m59.21916483s) [upload=335056]: Upload finished. err: <nil> Duration: 5.838334064s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X.....X..X....X........X.....................XXXX...X..X...] duration: 5.760568912s trace: https://jaeger.dev.storj.io/trace/2d9809728bd4cd0c FAIL (2025-12-21 13:18:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: GMCGUAPW66BGV34OH62E5Q66I27LMCCJSGYR5TUWJURFP77KVGJA) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: HMF7YEOHQ5GMCTBMEW3LBCDU2OEUL37IHAHKJTWFID2OUNULAZXQ) context deadline exceeded ======LOGS====== Dec 21 13:15:56.527259459 (381h46m56.398342504s) [upload=335056]: Starting upload Dec 21 13:15:56.527348678 (381h46m56.398431733s) [upload=335056]: Done starting upload Dec 21 13:15:56.527441445 (381h46m56.39852448s) [upload=335056]: Uploading segments... Dec 21 13:15:56.527575821 (381h46m56.398658856s) [upload=335056]: Got next segment. Inline: false Dec 21 13:15:56.528555026 (381h46m56.399638061s) [upload=335056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 13:15:56.528563231 (381h46m56.399646266s) [upload=335056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 13:15:56.775777978 (381h46m56.646861033s) [seg_pos={0 0}, upload=335056]: Begin upload segment... Dec 21 13:15:56.775835256 (381h46m56.646918291s) [seg_pos={0 0}, upload=335056]: Done begin upload segment. Dec 21 13:15:56.775875322 (381h46m56.646958378s) [noise=true , address=145.79.227.43:4157 , piece_id=FUCMIO5FINNCMGYXMDL3G7GY3H7XYQMXSV65F6ILKVWFDIK3BK6Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.775888899 (381h46m56.646971934s) [noise=true , address=45.91.116.14:4189 , piece_id=HMF7YEOHQ5GMCTBMEW3LBCDU2OEUL37IHAHKJTWFID2OUNULAZXQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.776387338 (381h46m56.647470383s) [noise=true , address=145.79.227.48:4301 , piece_id=LSMT6PC7K6BUFGXFFCNLN5TKMB4F7ARL24QRUGLPOO6RP5RNP7YA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.776605343 (381h46m56.647688378s) [noise=true , address=145.79.227.50:4375 , piece_id=U4WW6N3OB7G5T5BZXBYLFWMFERKOEA2VRF7JHRFMNZHKQRVF7U7Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.776799853 (381h46m56.647882898s) [noise=true , address=45.91.116.8:4046 , piece_id=5WFCBJKXGZX63HORQJDVDBDG46BMZ3VE7WWYZPEKXLWUKQ434RYQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777123059 (381h46m56.648206094s) [noise=true , address=145.79.227.46:4603 , piece_id=IBB2HKDX4UOF4OBD6OCPAELN7K5KRKBB3TNCAD4AOAAM3DWT6BDA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777207400 (381h46m56.648290445s) [noise=true , address=145.79.227.52:4445 , piece_id=H7BWJDSSISRYQGOBM5ZKVHR4FKMH5LTONG2T56WTX6IKH4IGIGQQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777533130 (381h46m56.648616165s) [noise=true , address=145.79.227.37:4565 , piece_id=GU35UYW7DUSW2QI7DM4H7GUILILZN2BGPKR5SSPAYC7MTFYI5I4Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777602983 (381h46m56.648686018s) [noise=true , address=45.91.116.13:4156 , piece_id=GMCGUAPW66BGV34OH62E5Q66I27LMCCJSGYR5TUWJURFP77KVGJA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777919996 (381h46m56.649003111s) [noise=true , address=145.79.227.38:4979 , piece_id=A3SZK4HNWGB3AGYKDXVT57CAP5QPEXCJ2OZHIC7XEOEQPNHPV77A, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778138662 (381h46m56.649221697s) [noise=true , address=145.79.227.8:4330 , piece_id=O3BAAICZCQULVD6TSBPOGEHC7FNAB53QNL26OYVX5ED6ESWU64PQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778531932 (381h46m56.649614967s) [noise=true , address=45.91.116.10:4084 , piece_id=7X5ACNCEMTSKIVIDJQ72IPMJGSFC47WO6VN5JANFEMYIUDUZ6ALQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778556999 (381h46m56.649640044s) [noise=true , address=145.79.227.39:4013 , piece_id=GP6DMTAQHDHF2JTUVD6NWPWCS4F3LFMSDGKFK2TUTWQ7U6MXXB5Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778969454 (381h46m56.650052499s) [noise=true , address=77.90.31.10:4519 , piece_id=YWYV2OZRPNYCRFZUXU7I4NPSDZ4TPBPDKZLZTFDP3Z2T6VBD77HA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.779064235 (381h46m56.65014728s) [noise=true , address=145.79.227.29:4943 , piece_id=AR2GVI4J2Q5JY7JE7G7ZYFRNYTW6HHAJEA5AUJTUNEBL6SYEUVUA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.779624251 (381h46m56.650707296s) [noise=true , address=145.79.227.45:4230 , piece_id=WOKT7BUVBTCQMPKTNBWI5IBEVVBCGLJSWVBXPD3DSRXWXS5ZODAA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:58.473831251 (381h46m58.344914296s) [upload=335056]: Next returned nil segment Dec 21 13:15:58.473877700 (381h46m58.344960735s) [upload=335056]: Waiting for error group managing segments... Dec 21 13:15:58.750808791 (381h46m58.621891826s) [noise=true , address=145.79.227.8:4330 , piece_id=O3BAAICZCQULVD6TSBPOGEHC7FNAB53QNL26OYVX5ED6ESWU64PQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.760307128 (381h46m58.631390163s) [noise=true , address=145.79.227.37:4565 , piece_id=GU35UYW7DUSW2QI7DM4H7GUILILZN2BGPKR5SSPAYC7MTFYI5I4Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.761544894 (381h46m58.632627929s) [noise=true , address=145.79.227.50:4375 , piece_id=U4WW6N3OB7G5T5BZXBYLFWMFERKOEA2VRF7JHRFMNZHKQRVF7U7Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.767805456 (381h46m58.638888501s) [noise=true , address=145.79.227.38:4979 , piece_id=A3SZK4HNWGB3AGYKDXVT57CAP5QPEXCJ2OZHIC7XEOEQPNHPV77A, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.770873438 (381h46m58.641956473s) [noise=true , address=145.79.227.48:4301 , piece_id=LSMT6PC7K6BUFGXFFCNLN5TKMB4F7ARL24QRUGLPOO6RP5RNP7YA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.771314767 (381h46m58.642397802s) [noise=true , address=145.79.227.43:4157 , piece_id=FUCMIO5FINNCMGYXMDL3G7GY3H7XYQMXSV65F6ILKVWFDIK3BK6Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.792102381 (381h46m58.663185416s) [noise=true , address=145.79.227.46:4603 , piece_id=IBB2HKDX4UOF4OBD6OCPAELN7K5KRKBB3TNCAD4AOAAM3DWT6BDA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.800571156 (381h46m58.671654191s) [noise=true , address=145.79.227.29:4943 , piece_id=AR2GVI4J2Q5JY7JE7G7ZYFRNYTW6HHAJEA5AUJTUNEBL6SYEUVUA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.827627135 (381h46m58.69871018s) [noise=true , address=145.79.227.52:4445 , piece_id=H7BWJDSSISRYQGOBM5ZKVHR4FKMH5LTONG2T56WTX6IKH4IGIGQQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.832837256 (381h46m58.703920291s) [noise=true , address=145.79.227.45:4230 , piece_id=WOKT7BUVBTCQMPKTNBWI5IBEVVBCGLJSWVBXPD3DSRXWXS5ZODAA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.856094943 (381h46m58.727177988s) [noise=true , address=145.79.227.39:4013 , piece_id=GP6DMTAQHDHF2JTUVD6NWPWCS4F3LFMSDGKFK2TUTWQ7U6MXXB5Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:59.008546104 (381h46m58.879629149s) [noise=true , address=45.91.116.13:4156 , piece_id=GMCGUAPW66BGV34OH62E5Q66I27LMCCJSGYR5TUWJURFP77KVGJA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:59.052840027 (381h46m58.923923062s) [noise=true , address=45.91.116.14:4189 , piece_id=HMF7YEOHQ5GMCTBMEW3LBCDU2OEUL37IHAHKJTWFID2OUNULAZXQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:59.052858953 (381h46m58.923941988s) [seg_pos={0 0}, upload=335056]: Segment reached optimal threshold of 13 pieces. Dec 21 13:15:59.053141110 (381h46m58.924224155s) [noise=true , address=45.91.116.10:4084 , piece_id=7X5ACNCEMTSKIVIDJQ72IPMJGSFC47WO6VN5JANFEMYIUDUZ6ALQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 21 13:15:59.053148975 (381h46m58.92423202s) [noise=true , address=45.91.116.10:4084 , piece_id=7X5ACNCEMTSKIVIDJQ72IPMJGSFC47WO6VN5JANFEMYIUDUZ6ALQ, seg_pos={0 0}, upload=335056]: Long tail cancellation due to optimal threshold Dec 21 13:15:59.053223627 (381h46m58.924306662s) [noise=true , address=45.91.116.8:4046 , piece_id=5WFCBJKXGZX63HORQJDVDBDG46BMZ3VE7WWYZPEKXLWUKQ434RYQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 21 13:15:59.053253414 (381h46m58.924336449s) [noise=true , address=45.91.116.8:4046 , piece_id=5WFCBJKXGZX63HORQJDVDBDG46BMZ3VE7WWYZPEKXLWUKQ434RYQ, seg_pos={0 0}, upload=335056]: Long tail cancellation due to optimal threshold Dec 21 13:15:59.053407067 (381h46m58.924490112s) [noise=true , address=77.90.31.10:4519 , piece_id=YWYV2OZRPNYCRFZUXU7I4NPSDZ4TPBPDKZLZTFDP3Z2T6VBD77HA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 21 13:15:59.053414541 (381h46m58.924497576s) [noise=true , address=77.90.31.10:4519 , piece_id=YWYV2OZRPNYCRFZUXU7I4NPSDZ4TPBPDKZLZTFDP3Z2T6VBD77HA, seg_pos={0 0}, upload=335056]: Long tail cancellation due to optimal threshold Dec 21 13:15:59.053429720 (381h46m58.924512765s) [seg_pos={0 0}, upload=335056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 13:15:59.053500063 (381h46m58.924583109s) [upload=335056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 13:15:59.053507238 (381h46m58.924590283s) [upload=335056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 13:15:59.348061896 (381h46m59.219144941s) [upload=335056]: Done uploading segments... Dec 21 13:15:59.348081795 (381h46m59.21916483s) [upload=335056]: Upload finished. err: <nil> Duration: 5.760598629s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..X.....X..X....X........X.....................XXXX...X..X..] duration: 5.974778684s trace: https://jaeger.dev.storj.io/trace/2d9809728bd4cd0c FAIL (2025-12-21 13:17:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: GMCGUAPW66BGV34OH62E5Q66I27LMCCJSGYR5TUWJURFP77KVGJA) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: HMF7YEOHQ5GMCTBMEW3LBCDU2OEUL37IHAHKJTWFID2OUNULAZXQ) context deadline exceeded ======LOGS====== Dec 21 13:15:56.527259459 (381h46m56.398342504s) [upload=335056]: Starting upload Dec 21 13:15:56.527348678 (381h46m56.398431733s) [upload=335056]: Done starting upload Dec 21 13:15:56.527441445 (381h46m56.39852448s) [upload=335056]: Uploading segments... Dec 21 13:15:56.527575821 (381h46m56.398658856s) [upload=335056]: Got next segment. Inline: false Dec 21 13:15:56.528555026 (381h46m56.399638061s) [upload=335056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 13:15:56.528563231 (381h46m56.399646266s) [upload=335056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 13:15:56.775777978 (381h46m56.646861033s) [seg_pos={0 0}, upload=335056]: Begin upload segment... Dec 21 13:15:56.775835256 (381h46m56.646918291s) [seg_pos={0 0}, upload=335056]: Done begin upload segment. Dec 21 13:15:56.775875322 (381h46m56.646958378s) [noise=true , address=145.79.227.43:4157 , piece_id=FUCMIO5FINNCMGYXMDL3G7GY3H7XYQMXSV65F6ILKVWFDIK3BK6Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.775888899 (381h46m56.646971934s) [noise=true , address=45.91.116.14:4189 , piece_id=HMF7YEOHQ5GMCTBMEW3LBCDU2OEUL37IHAHKJTWFID2OUNULAZXQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.776387338 (381h46m56.647470383s) [noise=true , address=145.79.227.48:4301 , piece_id=LSMT6PC7K6BUFGXFFCNLN5TKMB4F7ARL24QRUGLPOO6RP5RNP7YA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.776605343 (381h46m56.647688378s) [noise=true , address=145.79.227.50:4375 , piece_id=U4WW6N3OB7G5T5BZXBYLFWMFERKOEA2VRF7JHRFMNZHKQRVF7U7Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.776799853 (381h46m56.647882898s) [noise=true , address=45.91.116.8:4046 , piece_id=5WFCBJKXGZX63HORQJDVDBDG46BMZ3VE7WWYZPEKXLWUKQ434RYQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777123059 (381h46m56.648206094s) [noise=true , address=145.79.227.46:4603 , piece_id=IBB2HKDX4UOF4OBD6OCPAELN7K5KRKBB3TNCAD4AOAAM3DWT6BDA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777207400 (381h46m56.648290445s) [noise=true , address=145.79.227.52:4445 , piece_id=H7BWJDSSISRYQGOBM5ZKVHR4FKMH5LTONG2T56WTX6IKH4IGIGQQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777533130 (381h46m56.648616165s) [noise=true , address=145.79.227.37:4565 , piece_id=GU35UYW7DUSW2QI7DM4H7GUILILZN2BGPKR5SSPAYC7MTFYI5I4Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777602983 (381h46m56.648686018s) [noise=true , address=45.91.116.13:4156 , piece_id=GMCGUAPW66BGV34OH62E5Q66I27LMCCJSGYR5TUWJURFP77KVGJA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777919996 (381h46m56.649003111s) [noise=true , address=145.79.227.38:4979 , piece_id=A3SZK4HNWGB3AGYKDXVT57CAP5QPEXCJ2OZHIC7XEOEQPNHPV77A, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778138662 (381h46m56.649221697s) [noise=true , address=145.79.227.8:4330 , piece_id=O3BAAICZCQULVD6TSBPOGEHC7FNAB53QNL26OYVX5ED6ESWU64PQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778531932 (381h46m56.649614967s) [noise=true , address=45.91.116.10:4084 , piece_id=7X5ACNCEMTSKIVIDJQ72IPMJGSFC47WO6VN5JANFEMYIUDUZ6ALQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778556999 (381h46m56.649640044s) [noise=true , address=145.79.227.39:4013 , piece_id=GP6DMTAQHDHF2JTUVD6NWPWCS4F3LFMSDGKFK2TUTWQ7U6MXXB5Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778969454 (381h46m56.650052499s) [noise=true , address=77.90.31.10:4519 , piece_id=YWYV2OZRPNYCRFZUXU7I4NPSDZ4TPBPDKZLZTFDP3Z2T6VBD77HA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.779064235 (381h46m56.65014728s) [noise=true , address=145.79.227.29:4943 , piece_id=AR2GVI4J2Q5JY7JE7G7ZYFRNYTW6HHAJEA5AUJTUNEBL6SYEUVUA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.779624251 (381h46m56.650707296s) [noise=true , address=145.79.227.45:4230 , piece_id=WOKT7BUVBTCQMPKTNBWI5IBEVVBCGLJSWVBXPD3DSRXWXS5ZODAA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:58.473831251 (381h46m58.344914296s) [upload=335056]: Next returned nil segment Dec 21 13:15:58.473877700 (381h46m58.344960735s) [upload=335056]: Waiting for error group managing segments... Dec 21 13:15:58.750808791 (381h46m58.621891826s) [noise=true , address=145.79.227.8:4330 , piece_id=O3BAAICZCQULVD6TSBPOGEHC7FNAB53QNL26OYVX5ED6ESWU64PQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.760307128 (381h46m58.631390163s) [noise=true , address=145.79.227.37:4565 , piece_id=GU35UYW7DUSW2QI7DM4H7GUILILZN2BGPKR5SSPAYC7MTFYI5I4Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.761544894 (381h46m58.632627929s) [noise=true , address=145.79.227.50:4375 , piece_id=U4WW6N3OB7G5T5BZXBYLFWMFERKOEA2VRF7JHRFMNZHKQRVF7U7Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.767805456 (381h46m58.638888501s) [noise=true , address=145.79.227.38:4979 , piece_id=A3SZK4HNWGB3AGYKDXVT57CAP5QPEXCJ2OZHIC7XEOEQPNHPV77A, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.770873438 (381h46m58.641956473s) [noise=true , address=145.79.227.48:4301 , piece_id=LSMT6PC7K6BUFGXFFCNLN5TKMB4F7ARL24QRUGLPOO6RP5RNP7YA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.771314767 (381h46m58.642397802s) [noise=true , address=145.79.227.43:4157 , piece_id=FUCMIO5FINNCMGYXMDL3G7GY3H7XYQMXSV65F6ILKVWFDIK3BK6Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.792102381 (381h46m58.663185416s) [noise=true , address=145.79.227.46:4603 , piece_id=IBB2HKDX4UOF4OBD6OCPAELN7K5KRKBB3TNCAD4AOAAM3DWT6BDA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.800571156 (381h46m58.671654191s) [noise=true , address=145.79.227.29:4943 , piece_id=AR2GVI4J2Q5JY7JE7G7ZYFRNYTW6HHAJEA5AUJTUNEBL6SYEUVUA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.827627135 (381h46m58.69871018s) [noise=true , address=145.79.227.52:4445 , piece_id=H7BWJDSSISRYQGOBM5ZKVHR4FKMH5LTONG2T56WTX6IKH4IGIGQQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.832837256 (381h46m58.703920291s) [noise=true , address=145.79.227.45:4230 , piece_id=WOKT7BUVBTCQMPKTNBWI5IBEVVBCGLJSWVBXPD3DSRXWXS5ZODAA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.856094943 (381h46m58.727177988s) [noise=true , address=145.79.227.39:4013 , piece_id=GP6DMTAQHDHF2JTUVD6NWPWCS4F3LFMSDGKFK2TUTWQ7U6MXXB5Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:59.008546104 (381h46m58.879629149s) [noise=true , address=45.91.116.13:4156 , piece_id=GMCGUAPW66BGV34OH62E5Q66I27LMCCJSGYR5TUWJURFP77KVGJA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:59.052840027 (381h46m58.923923062s) [noise=true , address=45.91.116.14:4189 , piece_id=HMF7YEOHQ5GMCTBMEW3LBCDU2OEUL37IHAHKJTWFID2OUNULAZXQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:59.052858953 (381h46m58.923941988s) [seg_pos={0 0}, upload=335056]: Segment reached optimal threshold of 13 pieces. Dec 21 13:15:59.053141110 (381h46m58.924224155s) [noise=true , address=45.91.116.10:4084 , piece_id=7X5ACNCEMTSKIVIDJQ72IPMJGSFC47WO6VN5JANFEMYIUDUZ6ALQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 21 13:15:59.053148975 (381h46m58.92423202s) [noise=true , address=45.91.116.10:4084 , piece_id=7X5ACNCEMTSKIVIDJQ72IPMJGSFC47WO6VN5JANFEMYIUDUZ6ALQ, seg_pos={0 0}, upload=335056]: Long tail cancellation due to optimal threshold Dec 21 13:15:59.053223627 (381h46m58.924306662s) [noise=true , address=45.91.116.8:4046 , piece_id=5WFCBJKXGZX63HORQJDVDBDG46BMZ3VE7WWYZPEKXLWUKQ434RYQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 21 13:15:59.053253414 (381h46m58.924336449s) [noise=true , address=45.91.116.8:4046 , piece_id=5WFCBJKXGZX63HORQJDVDBDG46BMZ3VE7WWYZPEKXLWUKQ434RYQ, seg_pos={0 0}, upload=335056]: Long tail cancellation due to optimal threshold Dec 21 13:15:59.053407067 (381h46m58.924490112s) [noise=true , address=77.90.31.10:4519 , piece_id=YWYV2OZRPNYCRFZUXU7I4NPSDZ4TPBPDKZLZTFDP3Z2T6VBD77HA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 21 13:15:59.053414541 (381h46m58.924497576s) [noise=true , address=77.90.31.10:4519 , piece_id=YWYV2OZRPNYCRFZUXU7I4NPSDZ4TPBPDKZLZTFDP3Z2T6VBD77HA, seg_pos={0 0}, upload=335056]: Long tail cancellation due to optimal threshold Dec 21 13:15:59.053429720 (381h46m58.924512765s) [seg_pos={0 0}, upload=335056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 13:15:59.053500063 (381h46m58.924583109s) [upload=335056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 13:15:59.053507238 (381h46m58.924590283s) [upload=335056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 13:15:59.348061896 (381h46m59.219144941s) [upload=335056]: Done uploading segments... Dec 21 13:15:59.348081795 (381h46m59.21916483s) [upload=335056]: Upload finished. err: <nil> Duration: 5.974805206s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X..X.....X..X....X........X.....................XXXX...X..X.] duration: 5.957248374s trace: https://jaeger.dev.storj.io/trace/2d9809728bd4cd0c FAIL (2025-12-21 13:16:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: GMCGUAPW66BGV34OH62E5Q66I27LMCCJSGYR5TUWJURFP77KVGJA) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: HMF7YEOHQ5GMCTBMEW3LBCDU2OEUL37IHAHKJTWFID2OUNULAZXQ) context deadline exceeded ======LOGS====== Dec 21 13:15:56.527259459 (381h46m56.398342504s) [upload=335056]: Starting upload Dec 21 13:15:56.527348678 (381h46m56.398431733s) [upload=335056]: Done starting upload Dec 21 13:15:56.527441445 (381h46m56.39852448s) [upload=335056]: Uploading segments... Dec 21 13:15:56.527575821 (381h46m56.398658856s) [upload=335056]: Got next segment. Inline: false Dec 21 13:15:56.528555026 (381h46m56.399638061s) [upload=335056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 13:15:56.528563231 (381h46m56.399646266s) [upload=335056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 13:15:56.775777978 (381h46m56.646861033s) [seg_pos={0 0}, upload=335056]: Begin upload segment... Dec 21 13:15:56.775835256 (381h46m56.646918291s) [seg_pos={0 0}, upload=335056]: Done begin upload segment. Dec 21 13:15:56.775875322 (381h46m56.646958378s) [noise=true , address=145.79.227.43:4157 , piece_id=FUCMIO5FINNCMGYXMDL3G7GY3H7XYQMXSV65F6ILKVWFDIK3BK6Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.775888899 (381h46m56.646971934s) [noise=true , address=45.91.116.14:4189 , piece_id=HMF7YEOHQ5GMCTBMEW3LBCDU2OEUL37IHAHKJTWFID2OUNULAZXQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.776387338 (381h46m56.647470383s) [noise=true , address=145.79.227.48:4301 , piece_id=LSMT6PC7K6BUFGXFFCNLN5TKMB4F7ARL24QRUGLPOO6RP5RNP7YA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.776605343 (381h46m56.647688378s) [noise=true , address=145.79.227.50:4375 , piece_id=U4WW6N3OB7G5T5BZXBYLFWMFERKOEA2VRF7JHRFMNZHKQRVF7U7Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.776799853 (381h46m56.647882898s) [noise=true , address=45.91.116.8:4046 , piece_id=5WFCBJKXGZX63HORQJDVDBDG46BMZ3VE7WWYZPEKXLWUKQ434RYQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777123059 (381h46m56.648206094s) [noise=true , address=145.79.227.46:4603 , piece_id=IBB2HKDX4UOF4OBD6OCPAELN7K5KRKBB3TNCAD4AOAAM3DWT6BDA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777207400 (381h46m56.648290445s) [noise=true , address=145.79.227.52:4445 , piece_id=H7BWJDSSISRYQGOBM5ZKVHR4FKMH5LTONG2T56WTX6IKH4IGIGQQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777533130 (381h46m56.648616165s) [noise=true , address=145.79.227.37:4565 , piece_id=GU35UYW7DUSW2QI7DM4H7GUILILZN2BGPKR5SSPAYC7MTFYI5I4Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777602983 (381h46m56.648686018s) [noise=true , address=45.91.116.13:4156 , piece_id=GMCGUAPW66BGV34OH62E5Q66I27LMCCJSGYR5TUWJURFP77KVGJA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777919996 (381h46m56.649003111s) [noise=true , address=145.79.227.38:4979 , piece_id=A3SZK4HNWGB3AGYKDXVT57CAP5QPEXCJ2OZHIC7XEOEQPNHPV77A, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778138662 (381h46m56.649221697s) [noise=true , address=145.79.227.8:4330 , piece_id=O3BAAICZCQULVD6TSBPOGEHC7FNAB53QNL26OYVX5ED6ESWU64PQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778531932 (381h46m56.649614967s) [noise=true , address=45.91.116.10:4084 , piece_id=7X5ACNCEMTSKIVIDJQ72IPMJGSFC47WO6VN5JANFEMYIUDUZ6ALQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778556999 (381h46m56.649640044s) [noise=true , address=145.79.227.39:4013 , piece_id=GP6DMTAQHDHF2JTUVD6NWPWCS4F3LFMSDGKFK2TUTWQ7U6MXXB5Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778969454 (381h46m56.650052499s) [noise=true , address=77.90.31.10:4519 , piece_id=YWYV2OZRPNYCRFZUXU7I4NPSDZ4TPBPDKZLZTFDP3Z2T6VBD77HA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.779064235 (381h46m56.65014728s) [noise=true , address=145.79.227.29:4943 , piece_id=AR2GVI4J2Q5JY7JE7G7ZYFRNYTW6HHAJEA5AUJTUNEBL6SYEUVUA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.779624251 (381h46m56.650707296s) [noise=true , address=145.79.227.45:4230 , piece_id=WOKT7BUVBTCQMPKTNBWI5IBEVVBCGLJSWVBXPD3DSRXWXS5ZODAA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:58.473831251 (381h46m58.344914296s) [upload=335056]: Next returned nil segment Dec 21 13:15:58.473877700 (381h46m58.344960735s) [upload=335056]: Waiting for error group managing segments... Dec 21 13:15:58.750808791 (381h46m58.621891826s) [noise=true , address=145.79.227.8:4330 , piece_id=O3BAAICZCQULVD6TSBPOGEHC7FNAB53QNL26OYVX5ED6ESWU64PQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.760307128 (381h46m58.631390163s) [noise=true , address=145.79.227.37:4565 , piece_id=GU35UYW7DUSW2QI7DM4H7GUILILZN2BGPKR5SSPAYC7MTFYI5I4Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.761544894 (381h46m58.632627929s) [noise=true , address=145.79.227.50:4375 , piece_id=U4WW6N3OB7G5T5BZXBYLFWMFERKOEA2VRF7JHRFMNZHKQRVF7U7Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.767805456 (381h46m58.638888501s) [noise=true , address=145.79.227.38:4979 , piece_id=A3SZK4HNWGB3AGYKDXVT57CAP5QPEXCJ2OZHIC7XEOEQPNHPV77A, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.770873438 (381h46m58.641956473s) [noise=true , address=145.79.227.48:4301 , piece_id=LSMT6PC7K6BUFGXFFCNLN5TKMB4F7ARL24QRUGLPOO6RP5RNP7YA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.771314767 (381h46m58.642397802s) [noise=true , address=145.79.227.43:4157 , piece_id=FUCMIO5FINNCMGYXMDL3G7GY3H7XYQMXSV65F6ILKVWFDIK3BK6Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.792102381 (381h46m58.663185416s) [noise=true , address=145.79.227.46:4603 , piece_id=IBB2HKDX4UOF4OBD6OCPAELN7K5KRKBB3TNCAD4AOAAM3DWT6BDA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.800571156 (381h46m58.671654191s) [noise=true , address=145.79.227.29:4943 , piece_id=AR2GVI4J2Q5JY7JE7G7ZYFRNYTW6HHAJEA5AUJTUNEBL6SYEUVUA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.827627135 (381h46m58.69871018s) [noise=true , address=145.79.227.52:4445 , piece_id=H7BWJDSSISRYQGOBM5ZKVHR4FKMH5LTONG2T56WTX6IKH4IGIGQQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.832837256 (381h46m58.703920291s) [noise=true , address=145.79.227.45:4230 , piece_id=WOKT7BUVBTCQMPKTNBWI5IBEVVBCGLJSWVBXPD3DSRXWXS5ZODAA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.856094943 (381h46m58.727177988s) [noise=true , address=145.79.227.39:4013 , piece_id=GP6DMTAQHDHF2JTUVD6NWPWCS4F3LFMSDGKFK2TUTWQ7U6MXXB5Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:59.008546104 (381h46m58.879629149s) [noise=true , address=45.91.116.13:4156 , piece_id=GMCGUAPW66BGV34OH62E5Q66I27LMCCJSGYR5TUWJURFP77KVGJA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:59.052840027 (381h46m58.923923062s) [noise=true , address=45.91.116.14:4189 , piece_id=HMF7YEOHQ5GMCTBMEW3LBCDU2OEUL37IHAHKJTWFID2OUNULAZXQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:59.052858953 (381h46m58.923941988s) [seg_pos={0 0}, upload=335056]: Segment reached optimal threshold of 13 pieces. Dec 21 13:15:59.053141110 (381h46m58.924224155s) [noise=true , address=45.91.116.10:4084 , piece_id=7X5ACNCEMTSKIVIDJQ72IPMJGSFC47WO6VN5JANFEMYIUDUZ6ALQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 21 13:15:59.053148975 (381h46m58.92423202s) [noise=true , address=45.91.116.10:4084 , piece_id=7X5ACNCEMTSKIVIDJQ72IPMJGSFC47WO6VN5JANFEMYIUDUZ6ALQ, seg_pos={0 0}, upload=335056]: Long tail cancellation due to optimal threshold Dec 21 13:15:59.053223627 (381h46m58.924306662s) [noise=true , address=45.91.116.8:4046 , piece_id=5WFCBJKXGZX63HORQJDVDBDG46BMZ3VE7WWYZPEKXLWUKQ434RYQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 21 13:15:59.053253414 (381h46m58.924336449s) [noise=true , address=45.91.116.8:4046 , piece_id=5WFCBJKXGZX63HORQJDVDBDG46BMZ3VE7WWYZPEKXLWUKQ434RYQ, seg_pos={0 0}, upload=335056]: Long tail cancellation due to optimal threshold Dec 21 13:15:59.053407067 (381h46m58.924490112s) [noise=true , address=77.90.31.10:4519 , piece_id=YWYV2OZRPNYCRFZUXU7I4NPSDZ4TPBPDKZLZTFDP3Z2T6VBD77HA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 21 13:15:59.053414541 (381h46m58.924497576s) [noise=true , address=77.90.31.10:4519 , piece_id=YWYV2OZRPNYCRFZUXU7I4NPSDZ4TPBPDKZLZTFDP3Z2T6VBD77HA, seg_pos={0 0}, upload=335056]: Long tail cancellation due to optimal threshold Dec 21 13:15:59.053429720 (381h46m58.924512765s) [seg_pos={0 0}, upload=335056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 13:15:59.053500063 (381h46m58.924583109s) [upload=335056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 13:15:59.053507238 (381h46m58.924590283s) [upload=335056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 13:15:59.348061896 (381h46m59.219144941s) [upload=335056]: Done uploading segments... Dec 21 13:15:59.348081795 (381h46m59.21916483s) [upload=335056]: Upload finished. err: <nil> Duration: 5.957267971s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.X..X.....X..X....X........X.....................XXXX...X..X] duration: 30.003209163s trace: https://jaeger.dev.storj.io/trace/2d9809728bd4cd0c FAIL (2025-12-21 13:16:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: GMCGUAPW66BGV34OH62E5Q66I27LMCCJSGYR5TUWJURFP77KVGJA) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: HMF7YEOHQ5GMCTBMEW3LBCDU2OEUL37IHAHKJTWFID2OUNULAZXQ) context deadline exceeded ======LOGS====== Dec 21 13:15:56.527259459 (381h46m56.398342504s) [upload=335056]: Starting upload Dec 21 13:15:56.527348678 (381h46m56.398431733s) [upload=335056]: Done starting upload Dec 21 13:15:56.527441445 (381h46m56.39852448s) [upload=335056]: Uploading segments... Dec 21 13:15:56.527575821 (381h46m56.398658856s) [upload=335056]: Got next segment. Inline: false Dec 21 13:15:56.528555026 (381h46m56.399638061s) [upload=335056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 13:15:56.528563231 (381h46m56.399646266s) [upload=335056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 13:15:56.775777978 (381h46m56.646861033s) [seg_pos={0 0}, upload=335056]: Begin upload segment... Dec 21 13:15:56.775835256 (381h46m56.646918291s) [seg_pos={0 0}, upload=335056]: Done begin upload segment. Dec 21 13:15:56.775875322 (381h46m56.646958378s) [noise=true , address=145.79.227.43:4157 , piece_id=FUCMIO5FINNCMGYXMDL3G7GY3H7XYQMXSV65F6ILKVWFDIK3BK6Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.775888899 (381h46m56.646971934s) [noise=true , address=45.91.116.14:4189 , piece_id=HMF7YEOHQ5GMCTBMEW3LBCDU2OEUL37IHAHKJTWFID2OUNULAZXQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.776387338 (381h46m56.647470383s) [noise=true , address=145.79.227.48:4301 , piece_id=LSMT6PC7K6BUFGXFFCNLN5TKMB4F7ARL24QRUGLPOO6RP5RNP7YA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.776605343 (381h46m56.647688378s) [noise=true , address=145.79.227.50:4375 , piece_id=U4WW6N3OB7G5T5BZXBYLFWMFERKOEA2VRF7JHRFMNZHKQRVF7U7Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.776799853 (381h46m56.647882898s) [noise=true , address=45.91.116.8:4046 , piece_id=5WFCBJKXGZX63HORQJDVDBDG46BMZ3VE7WWYZPEKXLWUKQ434RYQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777123059 (381h46m56.648206094s) [noise=true , address=145.79.227.46:4603 , piece_id=IBB2HKDX4UOF4OBD6OCPAELN7K5KRKBB3TNCAD4AOAAM3DWT6BDA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777207400 (381h46m56.648290445s) [noise=true , address=145.79.227.52:4445 , piece_id=H7BWJDSSISRYQGOBM5ZKVHR4FKMH5LTONG2T56WTX6IKH4IGIGQQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777533130 (381h46m56.648616165s) [noise=true , address=145.79.227.37:4565 , piece_id=GU35UYW7DUSW2QI7DM4H7GUILILZN2BGPKR5SSPAYC7MTFYI5I4Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777602983 (381h46m56.648686018s) [noise=true , address=45.91.116.13:4156 , piece_id=GMCGUAPW66BGV34OH62E5Q66I27LMCCJSGYR5TUWJURFP77KVGJA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.777919996 (381h46m56.649003111s) [noise=true , address=145.79.227.38:4979 , piece_id=A3SZK4HNWGB3AGYKDXVT57CAP5QPEXCJ2OZHIC7XEOEQPNHPV77A, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778138662 (381h46m56.649221697s) [noise=true , address=145.79.227.8:4330 , piece_id=O3BAAICZCQULVD6TSBPOGEHC7FNAB53QNL26OYVX5ED6ESWU64PQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778531932 (381h46m56.649614967s) [noise=true , address=45.91.116.10:4084 , piece_id=7X5ACNCEMTSKIVIDJQ72IPMJGSFC47WO6VN5JANFEMYIUDUZ6ALQ, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778556999 (381h46m56.649640044s) [noise=true , address=145.79.227.39:4013 , piece_id=GP6DMTAQHDHF2JTUVD6NWPWCS4F3LFMSDGKFK2TUTWQ7U6MXXB5Q, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.778969454 (381h46m56.650052499s) [noise=true , address=77.90.31.10:4519 , piece_id=YWYV2OZRPNYCRFZUXU7I4NPSDZ4TPBPDKZLZTFDP3Z2T6VBD77HA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.779064235 (381h46m56.65014728s) [noise=true , address=145.79.227.29:4943 , piece_id=AR2GVI4J2Q5JY7JE7G7ZYFRNYTW6HHAJEA5AUJTUNEBL6SYEUVUA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:56.779624251 (381h46m56.650707296s) [noise=true , address=145.79.227.45:4230 , piece_id=WOKT7BUVBTCQMPKTNBWI5IBEVVBCGLJSWVBXPD3DSRXWXS5ZODAA, seg_pos={0 0}, upload=335056]: Uploading piece... Dec 21 13:15:58.473831251 (381h46m58.344914296s) [upload=335056]: Next returned nil segment Dec 21 13:15:58.473877700 (381h46m58.344960735s) [upload=335056]: Waiting for error group managing segments... Dec 21 13:15:58.750808791 (381h46m58.621891826s) [noise=true , address=145.79.227.8:4330 , piece_id=O3BAAICZCQULVD6TSBPOGEHC7FNAB53QNL26OYVX5ED6ESWU64PQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.760307128 (381h46m58.631390163s) [noise=true , address=145.79.227.37:4565 , piece_id=GU35UYW7DUSW2QI7DM4H7GUILILZN2BGPKR5SSPAYC7MTFYI5I4Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.761544894 (381h46m58.632627929s) [noise=true , address=145.79.227.50:4375 , piece_id=U4WW6N3OB7G5T5BZXBYLFWMFERKOEA2VRF7JHRFMNZHKQRVF7U7Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.767805456 (381h46m58.638888501s) [noise=true , address=145.79.227.38:4979 , piece_id=A3SZK4HNWGB3AGYKDXVT57CAP5QPEXCJ2OZHIC7XEOEQPNHPV77A, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.770873438 (381h46m58.641956473s) [noise=true , address=145.79.227.48:4301 , piece_id=LSMT6PC7K6BUFGXFFCNLN5TKMB4F7ARL24QRUGLPOO6RP5RNP7YA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.771314767 (381h46m58.642397802s) [noise=true , address=145.79.227.43:4157 , piece_id=FUCMIO5FINNCMGYXMDL3G7GY3H7XYQMXSV65F6ILKVWFDIK3BK6Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.792102381 (381h46m58.663185416s) [noise=true , address=145.79.227.46:4603 , piece_id=IBB2HKDX4UOF4OBD6OCPAELN7K5KRKBB3TNCAD4AOAAM3DWT6BDA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.800571156 (381h46m58.671654191s) [noise=true , address=145.79.227.29:4943 , piece_id=AR2GVI4J2Q5JY7JE7G7ZYFRNYTW6HHAJEA5AUJTUNEBL6SYEUVUA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.827627135 (381h46m58.69871018s) [noise=true , address=145.79.227.52:4445 , piece_id=H7BWJDSSISRYQGOBM5ZKVHR4FKMH5LTONG2T56WTX6IKH4IGIGQQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.832837256 (381h46m58.703920291s) [noise=true , address=145.79.227.45:4230 , piece_id=WOKT7BUVBTCQMPKTNBWI5IBEVVBCGLJSWVBXPD3DSRXWXS5ZODAA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:58.856094943 (381h46m58.727177988s) [noise=true , address=145.79.227.39:4013 , piece_id=GP6DMTAQHDHF2JTUVD6NWPWCS4F3LFMSDGKFK2TUTWQ7U6MXXB5Q, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:59.008546104 (381h46m58.879629149s) [noise=true , address=45.91.116.13:4156 , piece_id=GMCGUAPW66BGV34OH62E5Q66I27LMCCJSGYR5TUWJURFP77KVGJA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:59.052840027 (381h46m58.923923062s) [noise=true , address=45.91.116.14:4189 , piece_id=HMF7YEOHQ5GMCTBMEW3LBCDU2OEUL37IHAHKJTWFID2OUNULAZXQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: <nil> Dec 21 13:15:59.052858953 (381h46m58.923941988s) [seg_pos={0 0}, upload=335056]: Segment reached optimal threshold of 13 pieces. Dec 21 13:15:59.053141110 (381h46m58.924224155s) [noise=true , address=45.91.116.10:4084 , piece_id=7X5ACNCEMTSKIVIDJQ72IPMJGSFC47WO6VN5JANFEMYIUDUZ6ALQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 21 13:15:59.053148975 (381h46m58.92423202s) [noise=true , address=45.91.116.10:4084 , piece_id=7X5ACNCEMTSKIVIDJQ72IPMJGSFC47WO6VN5JANFEMYIUDUZ6ALQ, seg_pos={0 0}, upload=335056]: Long tail cancellation due to optimal threshold Dec 21 13:15:59.053223627 (381h46m58.924306662s) [noise=true , address=45.91.116.8:4046 , piece_id=5WFCBJKXGZX63HORQJDVDBDG46BMZ3VE7WWYZPEKXLWUKQ434RYQ, seg_pos={0 0}, upload=335056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 21 13:15:59.053253414 (381h46m58.924336449s) [noise=true , address=45.91.116.8:4046 , piece_id=5WFCBJKXGZX63HORQJDVDBDG46BMZ3VE7WWYZPEKXLWUKQ434RYQ, seg_pos={0 0}, upload=335056]: Long tail cancellation due to optimal threshold Dec 21 13:15:59.053407067 (381h46m58.924490112s) [noise=true , address=77.90.31.10:4519 , piece_id=YWYV2OZRPNYCRFZUXU7I4NPSDZ4TPBPDKZLZTFDP3Z2T6VBD77HA, seg_pos={0 0}, upload=335056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 21 13:15:59.053414541 (381h46m58.924497576s) [noise=true , address=77.90.31.10:4519 , piece_id=YWYV2OZRPNYCRFZUXU7I4NPSDZ4TPBPDKZLZTFDP3Z2T6VBD77HA, seg_pos={0 0}, upload=335056]: Long tail cancellation due to optimal threshold Dec 21 13:15:59.053429720 (381h46m58.924512765s) [seg_pos={0 0}, upload=335056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 13:15:59.053500063 (381h46m58.924583109s) [upload=335056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 13:15:59.053507238 (381h46m58.924590283s) [upload=335056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 13:15:59.348061896 (381h46m59.219144941s) [upload=335056]: Done uploading segments... Dec 21 13:15:59.348081795 (381h46m59.21916483s) [upload=335056]: Upload finished. err: <nil> Duration: 30.003232237s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..X..X.....X..X....X........X.....................XXXX...X..] duration: 5.915834837s trace: https://jaeger.dev.storj.io/trace/4813f976a9efc1d6 FAIL (2025-12-21 13:14:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: YCWPAPQN4747TQCST3SXBYG6KYGJHXPRCOLDZUTDN2J3KBKJ7G2A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: PDUQO7E2BBI4C73TFXHO3JYNFELEPJYSXD3IBQLFIVUWPXZB4I3A) context deadline exceeded ======LOGS====== Dec 21 13:13:27.135545255 (381h44m27.0066283s) [upload=335019]: Starting upload Dec 21 13:13:27.135639435 (381h44m27.00672248s) [upload=335019]: Done starting upload Dec 21 13:13:27.135891765 (381h44m27.0069748s) [upload=335019]: Uploading segments... Dec 21 13:13:27.135931100 (381h44m27.007014135s) [upload=335019]: Got next segment. Inline: false Dec 21 13:13:27.135976327 (381h44m27.007059362s) [upload=335019]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 13:13:27.135981015 (381h44m27.00706405s) [upload=335019]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 13:13:27.393633076 (381h44m27.264716111s) [seg_pos={0 0}, upload=335019]: Begin upload segment... Dec 21 13:13:27.393686918 (381h44m27.264769973s) [seg_pos={0 0}, upload=335019]: Done begin upload segment. Dec 21 13:13:27.393747243 (381h44m27.264830288s) [noise=true , address=145.79.227.15:4875 , piece_id=TP7J4NKDKY2D6D6DGLV6RJTC6XE7MEOFVQAKOLXNYWWCY26TD33Q, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:27.393864025 (381h44m27.26494707s) [noise=true , address=45.91.116.5:4300 , piece_id=PDUQO7E2BBI4C73TFXHO3JYNFELEPJYSXD3IBQLFIVUWPXZB4I3A, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:27.394571803 (381h44m27.265654849s) [noise=true , address=145.79.227.11:4731 , piece_id=MCOZ6FW76463XSNS5UC5NUEOOPCSNQ6IGTPYCQVYKNSE3ZXDDRUA, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:27.395297616 (381h44m27.266380661s) [noise=true , address=77.90.31.17:4770 , piece_id=WZDCUOXAXSLQDYFREWRJSEDHTRYTIGOF4SCJZCADM7AR5XOUGG6A, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:27.395642903 (381h44m27.266725938s) [noise=true , address=45.91.116.12:4228 , piece_id=SGUZFDN32NAJ7OMLC3SUVHTTUJRVEBMWNQWZ46O35DMDZORQV5WA, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:27.395970406 (381h44m27.267053452s) [noise=true , address=145.79.227.36:4518 , piece_id=5JHO5I24P54WIWWRBKMH5L7X4FSLMZLL3B2HGTESYKMUV5X6432A, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:27.396299903 (381h44m27.267382938s) [noise=true , address=45.91.116.6:4976 , piece_id=B2EGP2O5IP6LS3WGJ2PMZIJVSVD7VLEP6GQCJNZGM4YJNTDY75NA, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:27.396653317 (381h44m27.267736352s) [noise=true , address=145.79.227.44:4194 , piece_id=KRAURARXCK4G2U6XFW4BWQT2NRP5NTTVAYZ4JPL72O5M6JB5UZWA, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:27.397052868 (381h44m27.268135903s) [noise=true , address=145.79.227.47:4265 , piece_id=IN4BNZE3Q6S4WJDJH7ZPBCZ26QM7YUHCP5OVSYVRWB5T5C7PNI3Q, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:27.398608320 (381h44m27.269691365s) [noise=true , address=145.79.227.33:4869 , piece_id=D4CRS2GXVZ2RH2KSITZQAH22PKWUP4FXH4TQZYVG5LJDZTKWZJNA, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:27.398651712 (381h44m27.269734747s) [noise=true , address=145.79.227.52:4445 , piece_id=JELDYTT2NWHEIHQWQJB73AQ7RJ67C32MKJ6LVQ2URH27XDH6XP2Q, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:27.399881313 (381h44m27.270964358s) [noise=true , address=145.79.227.39:4015 , piece_id=26QJVNFCV2437ADPOUVZJAGWQGRGX2YD2UBAMUIZYYOFVPWYUU6A, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:27.400423526 (381h44m27.271506571s) [noise=true , address=45.91.116.9:4263 , piece_id=YCWPAPQN4747TQCST3SXBYG6KYGJHXPRCOLDZUTDN2J3KBKJ7G2A, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:27.401553918 (381h44m27.272636953s) [noise=true , address=145.79.227.38:4978 , piece_id=K3VNQZQBGLJYZNBIJHJDWAS4NUAN2OJBYUEX3KAZXIEGH6MJRF2Q, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:27.402023433 (381h44m27.273106468s) [noise=true , address=77.90.31.16:4736 , piece_id=4T6EX3IASMGJB4AKXTAN3MWY2J25QYBGBK5VEKAM6BSAPM5RCOHQ, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:27.403178442 (381h44m27.274261477s) [noise=true , address=145.79.227.41:4086 , piece_id=ZPAW4OK6GVI4FYY6ENSY2QVXLID76U44NTK6DJLF35IRGREQDRWQ, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:29.107290764 (381h44m28.978373799s) [upload=335019]: Next returned nil segment Dec 21 13:13:29.107307506 (381h44m28.978390541s) [upload=335019]: Waiting for error group managing segments... Dec 21 13:13:29.380433311 (381h44m29.251516346s) [noise=true , address=145.79.227.11:4731 , piece_id=MCOZ6FW76463XSNS5UC5NUEOOPCSNQ6IGTPYCQVYKNSE3ZXDDRUA, seg_pos={0 0}, upload=335019]: Done uploading piece. err: <nil> Dec 21 13:13:29.402434765 (381h44m29.27351781s) [noise=true , address=145.79.227.38:4978 , piece_id=K3VNQZQBGLJYZNBIJHJDWAS4NUAN2OJBYUEX3KAZXIEGH6MJRF2Q, seg_pos={0 0}, upload=335019]: Done uploading piece. err: <nil> Dec 21 13:13:29.416478487 (381h44m29.287561532s) [noise=true , address=145.79.227.41:4086 , piece_id=ZPAW4OK6GVI4FYY6ENSY2QVXLID76U44NTK6DJLF35IRGREQDRWQ, seg_pos={0 0}, upload=335019]: Done uploading piece. err: <nil> Dec 21 13:13:29.416716120 (381h44m29.287799165s) [noise=true , address=145.79.227.39:4015 , piece_id=26QJVNFCV2437ADPOUVZJAGWQGRGX2YD2UBAMUIZYYOFVPWYUU6A, seg_pos={0 0}, upload=335019]: Done uploading piece. err: <nil> Dec 21 13:13:29.420806959 (381h44m29.291890004s) [noise=true , address=145.79.227.33:4869 , piece_id=D4CRS2GXVZ2RH2KSITZQAH22PKWUP4FXH4TQZYVG5LJDZTKWZJNA, seg_pos={0 0}, upload=335019]: Done uploading piece. err: <nil> Dec 21 13:13:29.424254153 (381h44m29.295337198s) [noise=true , address=145.79.227.52:4445 , piece_id=JELDYTT2NWHEIHQWQJB73AQ7RJ67C32MKJ6LVQ2URH27XDH6XP2Q, seg_pos={0 0}, upload=335019]: Done uploading piece. err: <nil> Dec 21 13:13:29.440201690 (381h44m29.311284735s) [noise=true , address=145.79.227.15:4875 , piece_id=TP7J4NKDKY2D6D6DGLV6RJTC6XE7MEOFVQAKOLXNYWWCY26TD33Q, seg_pos={0 0}, upload=335019]: Done uploading piece. err: <nil> Dec 21 13:13:29.462338020 (381h44m29.333421075s) [noise=true , address=145.79.227.36:4518 , piece_id=5JHO5I24P54WIWWRBKMH5L7X4FSLMZLL3B2HGTESYKMUV5X6432A, seg_pos={0 0}, upload=335019]: Done uploading piece. err: <nil> Dec 21 13:13:29.477716674 (381h44m29.348799709s) [noise=true , address=145.79.227.47:4265 , piece_id=IN4BNZE3Q6S4WJDJH7ZPBCZ26QM7YUHCP5OVSYVRWB5T5C7PNI3Q, seg_pos={0 0}, upload=335019]: Done uploading piece. err: <nil> Dec 21 13:13:29.651138267 (381h44m29.522221302s) [noise=true , address=45.91.116.5:4300 , piece_id=PDUQO7E2BBI4C73TFXHO3JYNFELEPJYSXD3IBQLFIVUWPXZB4I3A, seg_pos={0 0}, upload=335019]: Done uploading piece. err: <nil> Dec 21 13:13:29.656902061 (381h44m29.527985097s) [noise=true , address=77.90.31.16:4736 , piece_id=4T6EX3IASMGJB4AKXTAN3MWY2J25QYBGBK5VEKAM6BSAPM5RCOHQ, seg_pos={0 0}, upload=335019]: Done uploading piece. err: <nil> Dec 21 13:13:29.709097141 (381h44m29.580180186s) [noise=true , address=45.91.116.9:4263 , piece_id=YCWPAPQN4747TQCST3SXBYG6KYGJHXPRCOLDZUTDN2J3KBKJ7G2A, seg_pos={0 0}, upload=335019]: Done uploading piece. err: <nil> Dec 21 13:13:29.712801624 (381h44m29.583884659s) [noise=true , address=45.91.116.6:4976 , piece_id=B2EGP2O5IP6LS3WGJ2PMZIJVSVD7VLEP6GQCJNZGM4YJNTDY75NA, seg_pos={0 0}, upload=335019]: Done uploading piece. err: <nil> Dec 21 13:13:29.712816442 (381h44m29.583899487s) [seg_pos={0 0}, upload=335019]: Segment reached optimal threshold of 13 pieces. Dec 21 13:13:29.712967590 (381h44m29.584050625s) [noise=true , address=145.79.227.44:4194 , piece_id=KRAURARXCK4G2U6XFW4BWQT2NRP5NTTVAYZ4JPL72O5M6JB5UZWA, seg_pos={0 0}, upload=335019]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 21 13:13:29.712974854 (381h44m29.584057889s) [noise=true , address=145.79.227.44:4194 , piece_id=KRAURARXCK4G2U6XFW4BWQT2NRP5NTTVAYZ4JPL72O5M6JB5UZWA, seg_pos={0 0}, upload=335019]: Long tail cancellation due to optimal threshold Dec 21 13:13:29.713228116 (381h44m29.584311151s) [noise=true , address=77.90.31.17:4770 , piece_id=WZDCUOXAXSLQDYFREWRJSEDHTRYTIGOF4SCJZCADM7AR5XOUGG6A, seg_pos={0 0}, upload=335019]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 21 13:13:29.713234598 (381h44m29.584317643s) [noise=true , address=77.90.31.17:4770 , piece_id=WZDCUOXAXSLQDYFREWRJSEDHTRYTIGOF4SCJZCADM7AR5XOUGG6A, seg_pos={0 0}, upload=335019]: Long tail cancellation due to optimal threshold Dec 21 13:13:29.713334559 (381h44m29.584417594s) [noise=true , address=45.91.116.12:4228 , piece_id=SGUZFDN32NAJ7OMLC3SUVHTTUJRVEBMWNQWZ46O35DMDZORQV5WA, seg_pos={0 0}, upload=335019]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 21 13:13:29.713339718 (381h44m29.584422753s) [noise=true , address=45.91.116.12:4228 , piece_id=SGUZFDN32NAJ7OMLC3SUVHTTUJRVEBMWNQWZ46O35DMDZORQV5WA, seg_pos={0 0}, upload=335019]: Long tail cancellation due to optimal threshold Dec 21 13:13:29.713356421 (381h44m29.584439456s) [seg_pos={0 0}, upload=335019]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 13:13:29.713422106 (381h44m29.584505141s) [upload=335019]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 13:13:29.713427235 (381h44m29.58451027s) [upload=335019]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 13:13:30.010896744 (381h44m29.881979779s) [upload=335019]: Done uploading segments... Dec 21 13:13:30.010928525 (381h44m29.88201156s) [upload=335019]: Upload finished. err: <nil> Duration: 5.915859976s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...X..X.....X..X....X........X.....................XXXX...X.] duration: 6.624277593s trace: https://jaeger.dev.storj.io/trace/4813f976a9efc1d6 FAIL (2025-12-21 13:14:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: YCWPAPQN4747TQCST3SXBYG6KYGJHXPRCOLDZUTDN2J3KBKJ7G2A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: PDUQO7E2BBI4C73TFXHO3JYNFELEPJYSXD3IBQLFIVUWPXZB4I3A) context deadline exceeded ======LOGS====== Dec 21 13:13:27.135545255 (381h44m27.0066283s) [upload=335019]: Starting upload Dec 21 13:13:27.135639435 (381h44m27.00672248s) [upload=335019]: Done starting upload Dec 21 13:13:27.135891765 (381h44m27.0069748s) [upload=335019]: Uploading segments... Dec 21 13:13:27.135931100 (381h44m27.007014135s) [upload=335019]: Got next segment. Inline: false Dec 21 13:13:27.135976327 (381h44m27.007059362s) [upload=335019]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 13:13:27.135981015 (381h44m27.00706405s) [upload=335019]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 13:13:27.393633076 (381h44m27.264716111s) [seg_pos={0 0}, upload=335019]: Begin upload segment... Dec 21 13:13:27.393686918 (381h44m27.264769973s) [seg_pos={0 0}, upload=335019]: Done begin upload segment. Dec 21 13:13:27.393747243 (381h44m27.264830288s) [noise=true , address=145.79.227.15:4875 , piece_id=TP7J4NKDKY2D6D6DGLV6RJTC6XE7MEOFVQAKOLXNYWWCY26TD33Q, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:27.393864025 (381h44m27.26494707s) [noise=true , address=45.91.116.5:4300 , piece_id=PDUQO7E2BBI4C73TFXHO3JYNFELEPJYSXD3IBQLFIVUWPXZB4I3A, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:27.394571803 (381h44m27.265654849s) [noise=true , address=145.79.227.11:4731 , piece_id=MCOZ6FW76463XSNS5UC5NUEOOPCSNQ6IGTPYCQVYKNSE3ZXDDRUA, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:27.395297616 (381h44m27.266380661s) [noise=true , address=77.90.31.17:4770 , piece_id=WZDCUOXAXSLQDYFREWRJSEDHTRYTIGOF4SCJZCADM7AR5XOUGG6A, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:27.395642903 (381h44m27.266725938s) [noise=true , address=45.91.116.12:4228 , piece_id=SGUZFDN32NAJ7OMLC3SUVHTTUJRVEBMWNQWZ46O35DMDZORQV5WA, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:27.395970406 (381h44m27.267053452s) [noise=true , address=145.79.227.36:4518 , piece_id=5JHO5I24P54WIWWRBKMH5L7X4FSLMZLL3B2HGTESYKMUV5X6432A, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:27.396299903 (381h44m27.267382938s) [noise=true , address=45.91.116.6:4976 , piece_id=B2EGP2O5IP6LS3WGJ2PMZIJVSVD7VLEP6GQCJNZGM4YJNTDY75NA, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:27.396653317 (381h44m27.267736352s) [noise=true , address=145.79.227.44:4194 , piece_id=KRAURARXCK4G2U6XFW4BWQT2NRP5NTTVAYZ4JPL72O5M6JB5UZWA, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:27.397052868 (381h44m27.268135903s) [noise=true , address=145.79.227.47:4265 , piece_id=IN4BNZE3Q6S4WJDJH7ZPBCZ26QM7YUHCP5OVSYVRWB5T5C7PNI3Q, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:27.398608320 (381h44m27.269691365s) [noise=true , address=145.79.227.33:4869 , piece_id=D4CRS2GXVZ2RH2KSITZQAH22PKWUP4FXH4TQZYVG5LJDZTKWZJNA, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:27.398651712 (381h44m27.269734747s) [noise=true , address=145.79.227.52:4445 , piece_id=JELDYTT2NWHEIHQWQJB73AQ7RJ67C32MKJ6LVQ2URH27XDH6XP2Q, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:27.399881313 (381h44m27.270964358s) [noise=true , address=145.79.227.39:4015 , piece_id=26QJVNFCV2437ADPOUVZJAGWQGRGX2YD2UBAMUIZYYOFVPWYUU6A, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:27.400423526 (381h44m27.271506571s) [noise=true , address=45.91.116.9:4263 , piece_id=YCWPAPQN4747TQCST3SXBYG6KYGJHXPRCOLDZUTDN2J3KBKJ7G2A, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:27.401553918 (381h44m27.272636953s) [noise=true , address=145.79.227.38:4978 , piece_id=K3VNQZQBGLJYZNBIJHJDWAS4NUAN2OJBYUEX3KAZXIEGH6MJRF2Q, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:27.402023433 (381h44m27.273106468s) [noise=true , address=77.90.31.16:4736 , piece_id=4T6EX3IASMGJB4AKXTAN3MWY2J25QYBGBK5VEKAM6BSAPM5RCOHQ, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:27.403178442 (381h44m27.274261477s) [noise=true , address=145.79.227.41:4086 , piece_id=ZPAW4OK6GVI4FYY6ENSY2QVXLID76U44NTK6DJLF35IRGREQDRWQ, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:29.107290764 (381h44m28.978373799s) [upload=335019]: Next returned nil segment Dec 21 13:13:29.107307506 (381h44m28.978390541s) [upload=335019]: Waiting for error group managing segments... Dec 21 13:13:29.380433311 (381h44m29.251516346s) [noise=true , address=145.79.227.11:4731 , piece_id=MCOZ6FW76463XSNS5UC5NUEOOPCSNQ6IGTPYCQVYKNSE3ZXDDRUA, seg_pos={0 0}, upload=335019]: Done uploading piece. err: <nil> Dec 21 13:13:29.402434765 (381h44m29.27351781s) [noise=true , address=145.79.227.38:4978 , piece_id=K3VNQZQBGLJYZNBIJHJDWAS4NUAN2OJBYUEX3KAZXIEGH6MJRF2Q, seg_pos={0 0}, upload=335019]: Done uploading piece. err: <nil> Dec 21 13:13:29.416478487 (381h44m29.287561532s) [noise=true , address=145.79.227.41:4086 , piece_id=ZPAW4OK6GVI4FYY6ENSY2QVXLID76U44NTK6DJLF35IRGREQDRWQ, seg_pos={0 0}, upload=335019]: Done uploading piece. err: <nil> Dec 21 13:13:29.416716120 (381h44m29.287799165s) [noise=true , address=145.79.227.39:4015 , piece_id=26QJVNFCV2437ADPOUVZJAGWQGRGX2YD2UBAMUIZYYOFVPWYUU6A, seg_pos={0 0}, upload=335019]: Done uploading piece. err: <nil> Dec 21 13:13:29.420806959 (381h44m29.291890004s) [noise=true , address=145.79.227.33:4869 , piece_id=D4CRS2GXVZ2RH2KSITZQAH22PKWUP4FXH4TQZYVG5LJDZTKWZJNA, seg_pos={0 0}, upload=335019]: Done uploading piece. err: <nil> Dec 21 13:13:29.424254153 (381h44m29.295337198s) [noise=true , address=145.79.227.52:4445 , piece_id=JELDYTT2NWHEIHQWQJB73AQ7RJ67C32MKJ6LVQ2URH27XDH6XP2Q, seg_pos={0 0}, upload=335019]: Done uploading piece. err: <nil> Dec 21 13:13:29.440201690 (381h44m29.311284735s) [noise=true , address=145.79.227.15:4875 , piece_id=TP7J4NKDKY2D6D6DGLV6RJTC6XE7MEOFVQAKOLXNYWWCY26TD33Q, seg_pos={0 0}, upload=335019]: Done uploading piece. err: <nil> Dec 21 13:13:29.462338020 (381h44m29.333421075s) [noise=true , address=145.79.227.36:4518 , piece_id=5JHO5I24P54WIWWRBKMH5L7X4FSLMZLL3B2HGTESYKMUV5X6432A, seg_pos={0 0}, upload=335019]: Done uploading piece. err: <nil> Dec 21 13:13:29.477716674 (381h44m29.348799709s) [noise=true , address=145.79.227.47:4265 , piece_id=IN4BNZE3Q6S4WJDJH7ZPBCZ26QM7YUHCP5OVSYVRWB5T5C7PNI3Q, seg_pos={0 0}, upload=335019]: Done uploading piece. err: <nil> Dec 21 13:13:29.651138267 (381h44m29.522221302s) [noise=true , address=45.91.116.5:4300 , piece_id=PDUQO7E2BBI4C73TFXHO3JYNFELEPJYSXD3IBQLFIVUWPXZB4I3A, seg_pos={0 0}, upload=335019]: Done uploading piece. err: <nil> Dec 21 13:13:29.656902061 (381h44m29.527985097s) [noise=true , address=77.90.31.16:4736 , piece_id=4T6EX3IASMGJB4AKXTAN3MWY2J25QYBGBK5VEKAM6BSAPM5RCOHQ, seg_pos={0 0}, upload=335019]: Done uploading piece. err: <nil> Dec 21 13:13:29.709097141 (381h44m29.580180186s) [noise=true , address=45.91.116.9:4263 , piece_id=YCWPAPQN4747TQCST3SXBYG6KYGJHXPRCOLDZUTDN2J3KBKJ7G2A, seg_pos={0 0}, upload=335019]: Done uploading piece. err: <nil> Dec 21 13:13:29.712801624 (381h44m29.583884659s) [noise=true , address=45.91.116.6:4976 , piece_id=B2EGP2O5IP6LS3WGJ2PMZIJVSVD7VLEP6GQCJNZGM4YJNTDY75NA, seg_pos={0 0}, upload=335019]: Done uploading piece. err: <nil> Dec 21 13:13:29.712816442 (381h44m29.583899487s) [seg_pos={0 0}, upload=335019]: Segment reached optimal threshold of 13 pieces. Dec 21 13:13:29.712967590 (381h44m29.584050625s) [noise=true , address=145.79.227.44:4194 , piece_id=KRAURARXCK4G2U6XFW4BWQT2NRP5NTTVAYZ4JPL72O5M6JB5UZWA, seg_pos={0 0}, upload=335019]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 21 13:13:29.712974854 (381h44m29.584057889s) [noise=true , address=145.79.227.44:4194 , piece_id=KRAURARXCK4G2U6XFW4BWQT2NRP5NTTVAYZ4JPL72O5M6JB5UZWA, seg_pos={0 0}, upload=335019]: Long tail cancellation due to optimal threshold Dec 21 13:13:29.713228116 (381h44m29.584311151s) [noise=true , address=77.90.31.17:4770 , piece_id=WZDCUOXAXSLQDYFREWRJSEDHTRYTIGOF4SCJZCADM7AR5XOUGG6A, seg_pos={0 0}, upload=335019]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 21 13:13:29.713234598 (381h44m29.584317643s) [noise=true , address=77.90.31.17:4770 , piece_id=WZDCUOXAXSLQDYFREWRJSEDHTRYTIGOF4SCJZCADM7AR5XOUGG6A, seg_pos={0 0}, upload=335019]: Long tail cancellation due to optimal threshold Dec 21 13:13:29.713334559 (381h44m29.584417594s) [noise=true , address=45.91.116.12:4228 , piece_id=SGUZFDN32NAJ7OMLC3SUVHTTUJRVEBMWNQWZ46O35DMDZORQV5WA, seg_pos={0 0}, upload=335019]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 21 13:13:29.713339718 (381h44m29.584422753s) [noise=true , address=45.91.116.12:4228 , piece_id=SGUZFDN32NAJ7OMLC3SUVHTTUJRVEBMWNQWZ46O35DMDZORQV5WA, seg_pos={0 0}, upload=335019]: Long tail cancellation due to optimal threshold Dec 21 13:13:29.713356421 (381h44m29.584439456s) [seg_pos={0 0}, upload=335019]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 13:13:29.713422106 (381h44m29.584505141s) [upload=335019]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 13:13:29.713427235 (381h44m29.58451027s) [upload=335019]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 13:13:30.010896744 (381h44m29.881979779s) [upload=335019]: Done uploading segments... Dec 21 13:13:30.010928525 (381h44m29.88201156s) [upload=335019]: Upload finished. err: <nil> Duration: 6.624297331s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [....X..X.....X..X....X........X.....................XXXX...X] duration: 30.003268194s trace: https://jaeger.dev.storj.io/trace/4813f976a9efc1d6 FAIL (2025-12-21 13:13:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: YCWPAPQN4747TQCST3SXBYG6KYGJHXPRCOLDZUTDN2J3KBKJ7G2A) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: PDUQO7E2BBI4C73TFXHO3JYNFELEPJYSXD3IBQLFIVUWPXZB4I3A) context deadline exceeded ======LOGS====== Dec 21 13:13:27.135545255 (381h44m27.0066283s) [upload=335019]: Starting upload Dec 21 13:13:27.135639435 (381h44m27.00672248s) [upload=335019]: Done starting upload Dec 21 13:13:27.135891765 (381h44m27.0069748s) [upload=335019]: Uploading segments... Dec 21 13:13:27.135931100 (381h44m27.007014135s) [upload=335019]: Got next segment. Inline: false Dec 21 13:13:27.135976327 (381h44m27.007059362s) [upload=335019]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 13:13:27.135981015 (381h44m27.00706405s) [upload=335019]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 13:13:27.393633076 (381h44m27.264716111s) [seg_pos={0 0}, upload=335019]: Begin upload segment... Dec 21 13:13:27.393686918 (381h44m27.264769973s) [seg_pos={0 0}, upload=335019]: Done begin upload segment. Dec 21 13:13:27.393747243 (381h44m27.264830288s) [noise=true , address=145.79.227.15:4875 , piece_id=TP7J4NKDKY2D6D6DGLV6RJTC6XE7MEOFVQAKOLXNYWWCY26TD33Q, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:27.393864025 (381h44m27.26494707s) [noise=true , address=45.91.116.5:4300 , piece_id=PDUQO7E2BBI4C73TFXHO3JYNFELEPJYSXD3IBQLFIVUWPXZB4I3A, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:27.394571803 (381h44m27.265654849s) [noise=true , address=145.79.227.11:4731 , piece_id=MCOZ6FW76463XSNS5UC5NUEOOPCSNQ6IGTPYCQVYKNSE3ZXDDRUA, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:27.395297616 (381h44m27.266380661s) [noise=true , address=77.90.31.17:4770 , piece_id=WZDCUOXAXSLQDYFREWRJSEDHTRYTIGOF4SCJZCADM7AR5XOUGG6A, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:27.395642903 (381h44m27.266725938s) [noise=true , address=45.91.116.12:4228 , piece_id=SGUZFDN32NAJ7OMLC3SUVHTTUJRVEBMWNQWZ46O35DMDZORQV5WA, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:27.395970406 (381h44m27.267053452s) [noise=true , address=145.79.227.36:4518 , piece_id=5JHO5I24P54WIWWRBKMH5L7X4FSLMZLL3B2HGTESYKMUV5X6432A, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:27.396299903 (381h44m27.267382938s) [noise=true , address=45.91.116.6:4976 , piece_id=B2EGP2O5IP6LS3WGJ2PMZIJVSVD7VLEP6GQCJNZGM4YJNTDY75NA, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:27.396653317 (381h44m27.267736352s) [noise=true , address=145.79.227.44:4194 , piece_id=KRAURARXCK4G2U6XFW4BWQT2NRP5NTTVAYZ4JPL72O5M6JB5UZWA, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:27.397052868 (381h44m27.268135903s) [noise=true , address=145.79.227.47:4265 , piece_id=IN4BNZE3Q6S4WJDJH7ZPBCZ26QM7YUHCP5OVSYVRWB5T5C7PNI3Q, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:27.398608320 (381h44m27.269691365s) [noise=true , address=145.79.227.33:4869 , piece_id=D4CRS2GXVZ2RH2KSITZQAH22PKWUP4FXH4TQZYVG5LJDZTKWZJNA, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:27.398651712 (381h44m27.269734747s) [noise=true , address=145.79.227.52:4445 , piece_id=JELDYTT2NWHEIHQWQJB73AQ7RJ67C32MKJ6LVQ2URH27XDH6XP2Q, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:27.399881313 (381h44m27.270964358s) [noise=true , address=145.79.227.39:4015 , piece_id=26QJVNFCV2437ADPOUVZJAGWQGRGX2YD2UBAMUIZYYOFVPWYUU6A, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:27.400423526 (381h44m27.271506571s) [noise=true , address=45.91.116.9:4263 , piece_id=YCWPAPQN4747TQCST3SXBYG6KYGJHXPRCOLDZUTDN2J3KBKJ7G2A, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:27.401553918 (381h44m27.272636953s) [noise=true , address=145.79.227.38:4978 , piece_id=K3VNQZQBGLJYZNBIJHJDWAS4NUAN2OJBYUEX3KAZXIEGH6MJRF2Q, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:27.402023433 (381h44m27.273106468s) [noise=true , address=77.90.31.16:4736 , piece_id=4T6EX3IASMGJB4AKXTAN3MWY2J25QYBGBK5VEKAM6BSAPM5RCOHQ, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:27.403178442 (381h44m27.274261477s) [noise=true , address=145.79.227.41:4086 , piece_id=ZPAW4OK6GVI4FYY6ENSY2QVXLID76U44NTK6DJLF35IRGREQDRWQ, seg_pos={0 0}, upload=335019]: Uploading piece... Dec 21 13:13:29.107290764 (381h44m28.978373799s) [upload=335019]: Next returned nil segment Dec 21 13:13:29.107307506 (381h44m28.978390541s) [upload=335019]: Waiting for error group managing segments... Dec 21 13:13:29.380433311 (381h44m29.251516346s) [noise=true , address=145.79.227.11:4731 , piece_id=MCOZ6FW76463XSNS5UC5NUEOOPCSNQ6IGTPYCQVYKNSE3ZXDDRUA, seg_pos={0 0}, upload=335019]: Done uploading piece. err: <nil> Dec 21 13:13:29.402434765 (381h44m29.27351781s) [noise=true , address=145.79.227.38:4978 , piece_id=K3VNQZQBGLJYZNBIJHJDWAS4NUAN2OJBYUEX3KAZXIEGH6MJRF2Q, seg_pos={0 0}, upload=335019]: Done uploading piece. err: <nil> Dec 21 13:13:29.416478487 (381h44m29.287561532s) [noise=true , address=145.79.227.41:4086 , piece_id=ZPAW4OK6GVI4FYY6ENSY2QVXLID76U44NTK6DJLF35IRGREQDRWQ, seg_pos={0 0}, upload=335019]: Done uploading piece. err: <nil> Dec 21 13:13:29.416716120 (381h44m29.287799165s) [noise=true , address=145.79.227.39:4015 , piece_id=26QJVNFCV2437ADPOUVZJAGWQGRGX2YD2UBAMUIZYYOFVPWYUU6A, seg_pos={0 0}, upload=335019]: Done uploading piece. err: <nil> Dec 21 13:13:29.420806959 (381h44m29.291890004s) [noise=true , address=145.79.227.33:4869 , piece_id=D4CRS2GXVZ2RH2KSITZQAH22PKWUP4FXH4TQZYVG5LJDZTKWZJNA, seg_pos={0 0}, upload=335019]: Done uploading piece. err: <nil> Dec 21 13:13:29.424254153 (381h44m29.295337198s) [noise=true , address=145.79.227.52:4445 , piece_id=JELDYTT2NWHEIHQWQJB73AQ7RJ67C32MKJ6LVQ2URH27XDH6XP2Q, seg_pos={0 0}, upload=335019]: Done uploading piece. err: <nil> Dec 21 13:13:29.440201690 (381h44m29.311284735s) [noise=true , address=145.79.227.15:4875 , piece_id=TP7J4NKDKY2D6D6DGLV6RJTC6XE7MEOFVQAKOLXNYWWCY26TD33Q, seg_pos={0 0}, upload=335019]: Done uploading piece. err: <nil> Dec 21 13:13:29.462338020 (381h44m29.333421075s) [noise=true , address=145.79.227.36:4518 , piece_id=5JHO5I24P54WIWWRBKMH5L7X4FSLMZLL3B2HGTESYKMUV5X6432A, seg_pos={0 0}, upload=335019]: Done uploading piece. err: <nil> Dec 21 13:13:29.477716674 (381h44m29.348799709s) [noise=true , address=145.79.227.47:4265 , piece_id=IN4BNZE3Q6S4WJDJH7ZPBCZ26QM7YUHCP5OVSYVRWB5T5C7PNI3Q, seg_pos={0 0}, upload=335019]: Done uploading piece. err: <nil> Dec 21 13:13:29.651138267 (381h44m29.522221302s) [noise=true , address=45.91.116.5:4300 , piece_id=PDUQO7E2BBI4C73TFXHO3JYNFELEPJYSXD3IBQLFIVUWPXZB4I3A, seg_pos={0 0}, upload=335019]: Done uploading piece. err: <nil> Dec 21 13:13:29.656902061 (381h44m29.527985097s) [noise=true , address=77.90.31.16:4736 , piece_id=4T6EX3IASMGJB4AKXTAN3MWY2J25QYBGBK5VEKAM6BSAPM5RCOHQ, seg_pos={0 0}, upload=335019]: Done uploading piece. err: <nil> Dec 21 13:13:29.709097141 (381h44m29.580180186s) [noise=true , address=45.91.116.9:4263 , piece_id=YCWPAPQN4747TQCST3SXBYG6KYGJHXPRCOLDZUTDN2J3KBKJ7G2A, seg_pos={0 0}, upload=335019]: Done uploading piece. err: <nil> Dec 21 13:13:29.712801624 (381h44m29.583884659s) [noise=true , address=45.91.116.6:4976 , piece_id=B2EGP2O5IP6LS3WGJ2PMZIJVSVD7VLEP6GQCJNZGM4YJNTDY75NA, seg_pos={0 0}, upload=335019]: Done uploading piece. err: <nil> Dec 21 13:13:29.712816442 (381h44m29.583899487s) [seg_pos={0 0}, upload=335019]: Segment reached optimal threshold of 13 pieces. Dec 21 13:13:29.712967590 (381h44m29.584050625s) [noise=true , address=145.79.227.44:4194 , piece_id=KRAURARXCK4G2U6XFW4BWQT2NRP5NTTVAYZ4JPL72O5M6JB5UZWA, seg_pos={0 0}, upload=335019]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 21 13:13:29.712974854 (381h44m29.584057889s) [noise=true , address=145.79.227.44:4194 , piece_id=KRAURARXCK4G2U6XFW4BWQT2NRP5NTTVAYZ4JPL72O5M6JB5UZWA, seg_pos={0 0}, upload=335019]: Long tail cancellation due to optimal threshold Dec 21 13:13:29.713228116 (381h44m29.584311151s) [noise=true , address=77.90.31.17:4770 , piece_id=WZDCUOXAXSLQDYFREWRJSEDHTRYTIGOF4SCJZCADM7AR5XOUGG6A, seg_pos={0 0}, upload=335019]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 21 13:13:29.713234598 (381h44m29.584317643s) [noise=true , address=77.90.31.17:4770 , piece_id=WZDCUOXAXSLQDYFREWRJSEDHTRYTIGOF4SCJZCADM7AR5XOUGG6A, seg_pos={0 0}, upload=335019]: Long tail cancellation due to optimal threshold Dec 21 13:13:29.713334559 (381h44m29.584417594s) [noise=true , address=45.91.116.12:4228 , piece_id=SGUZFDN32NAJ7OMLC3SUVHTTUJRVEBMWNQWZ46O35DMDZORQV5WA, seg_pos={0 0}, upload=335019]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 21 13:13:29.713339718 (381h44m29.584422753s) [noise=true , address=45.91.116.12:4228 , piece_id=SGUZFDN32NAJ7OMLC3SUVHTTUJRVEBMWNQWZ46O35DMDZORQV5WA, seg_pos={0 0}, upload=335019]: Long tail cancellation due to optimal threshold Dec 21 13:13:29.713356421 (381h44m29.584439456s) [seg_pos={0 0}, upload=335019]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 13:13:29.713422106 (381h44m29.584505141s) [upload=335019]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 13:13:29.713427235 (381h44m29.58451027s) [upload=335019]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 13:13:30.010896744 (381h44m29.881979779s) [upload=335019]: Done uploading segments... Dec 21 13:13:30.010928525 (381h44m29.88201156s) [upload=335019]: Upload finished. err: <nil> Duration: 30.003287992s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.....X..X.....X..X....X........X.....................XXXX...] duration: 5.700324503s trace: https://jaeger.dev.storj.io/trace/2ecb840715b827b4 FAIL (2025-12-21 13:13:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: FY4CVGTSSU2KIFZTKXQLAZXAMZICM2K344TNQLJEES253AHRFC6A) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: MMTEPTB6J3NWMT4T3IHEAJAG5YFFPTI56ZVW2PLGQWQWXLW5VJGQ) context deadline exceeded ======LOGS====== Dec 21 13:11:10.074980265 (381h42m9.94606331s) [upload=334980]: Starting upload Dec 21 13:11:10.075064275 (381h42m9.94614732s) [upload=334980]: Done starting upload Dec 21 13:11:10.075209582 (381h42m9.946292637s) [upload=334980]: Uploading segments... Dec 21 13:11:10.075280346 (381h42m9.946363392s) [upload=334980]: Got next segment. Inline: false Dec 21 13:11:10.075315323 (381h42m9.946398368s) [upload=334980]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 13:11:10.075320783 (381h42m9.946403828s) [upload=334980]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 13:11:10.327558205 (381h42m10.19864126s) [seg_pos={0 0}, upload=334980]: Begin upload segment... Dec 21 13:11:10.327625774 (381h42m10.198708809s) [seg_pos={0 0}, upload=334980]: Done begin upload segment. Dec 21 13:11:10.327682642 (381h42m10.198765677s) [noise=true , address=45.91.116.13:4154 , piece_id=3H4LTV35TLTMZCRXSSR4MQZ36QT5B7Z4WE2MGNJLIX77KTMZ4L3A, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.327709283 (381h42m10.198792328s) [noise=true , address=145.79.227.15:4875 , piece_id=32EQTBBSDBPSJONWUXUT5GJTWPGT7ZKU7QJSJ4XGZBSDGTAIZFWA, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.328281392 (381h42m10.199364467s) [noise=true , address=145.79.227.10:4403 , piece_id=O36LV7JYDF52VNSBPPG6XRHPJ4NKQIPBDYNHVKKWGMX2RBWM325A, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.328785282 (381h42m10.199868317s) [noise=true , address=145.79.227.32:4834 , piece_id=DUT5AUZUUETGQ57XEP25MYFD5HNYBYIWG6WFWXEE7A3NKKGEQ3OQ, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.328808836 (381h42m10.199891891s) [noise=true , address=45.91.116.9:4261 , piece_id=FY4CVGTSSU2KIFZTKXQLAZXAMZICM2K344TNQLJEES253AHRFC6A, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.329264193 (381h42m10.200347228s) [noise=true , address=77.90.31.14:4663 , piece_id=MMTEPTB6J3NWMT4T3IHEAJAG5YFFPTI56ZVW2PLGQWQWXLW5VJGQ, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.329720652 (381h42m10.200803698s) [noise=true , address=145.79.227.42:4121 , piece_id=JDDCHIOAV2CZSSAEQAH3D5PCFFE2XQMLTJDOAZEFXIMQRVYA3NMA, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.330129040 (381h42m10.201212075s) [noise=true , address=145.79.227.51:4411 , piece_id=KDCIGUCYVQXFJ7EEKKLYZYMDCTQLNOO7CPX7B5QUS2QC5NUNJJBQ, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.330507000 (381h42m10.201590035s) [noise=true , address=45.91.116.10:4082 , piece_id=LU2I6MA2UHRK44YDFMJKHJXJ5PUEX2CCCGDUYPKOTUCVV4TDSAOA, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.331607545 (381h42m10.20269058s) [noise=true , address=77.90.31.12:4590 , piece_id=RXNC4MPRT7UBD27PM2AVGRC2HUJW3IQ5DKMYYPPFYEH44ZUTYUTA, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.332067642 (381h42m10.203150677s) [noise=true , address=145.79.227.36:4517 , piece_id=PKQPPU7ZZE4NHX6RV6NXMOVWCOUMV5XDFOCJ4K7OSY6IFGTJMZFQ, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.333358941 (381h42m10.204441976s) [noise=true , address=77.90.31.10:4517 , piece_id=HDGCMARA4JQEL4M45A3ASVLT3EFKBTG4OKV3YM2GPS7JHJCESW3Q, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.336279552 (381h42m10.207362597s) [noise=true , address=145.79.227.37:4566 , piece_id=M5RUBUOWCO66UXO5BNCQT3I645D5NNUTYQPUD2Q57UO756ZSVIMA, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.342865683 (381h42m10.213948728s) [noise=true , address=145.79.227.48:4303 , piece_id=WVHYGAZ6GHV2P7NRITNGHDWWEUKV3M3NMORMN52V5YKEGU3QHPJA, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.345228553 (381h42m10.216311598s) [noise=true , address=145.79.227.25:4617 , piece_id=SGHIRIQNT2ESPG2TBWSUFXAL4MVHI67NCDSJOEL7QAOGQYYFSB4A, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.345748573 (381h42m10.216831618s) [noise=true , address=145.79.227.45:4231 , piece_id=YKLGXTWGMIDYHHXSI6SJWNWWANNLMI5IDS5ETBNXKYZQ5ULZJ3BA, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:12.050187306 (381h42m11.921270351s) [upload=334980]: Next returned nil segment Dec 21 13:11:12.050274632 (381h42m11.921357697s) [upload=334980]: Waiting for error group managing segments... Dec 21 13:11:12.333470151 (381h42m12.204553196s) [noise=true , address=145.79.227.51:4411 , piece_id=KDCIGUCYVQXFJ7EEKKLYZYMDCTQLNOO7CPX7B5QUS2QC5NUNJJBQ, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.334377990 (381h42m12.205461025s) [noise=true , address=145.79.227.32:4834 , piece_id=DUT5AUZUUETGQ57XEP25MYFD5HNYBYIWG6WFWXEE7A3NKKGEQ3OQ, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.338961147 (381h42m12.210044192s) [noise=true , address=145.79.227.10:4403 , piece_id=O36LV7JYDF52VNSBPPG6XRHPJ4NKQIPBDYNHVKKWGMX2RBWM325A, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.349152604 (381h42m12.220235649s) [noise=true , address=145.79.227.37:4566 , piece_id=M5RUBUOWCO66UXO5BNCQT3I645D5NNUTYQPUD2Q57UO756ZSVIMA, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.351128877 (381h42m12.222211922s) [noise=true , address=145.79.227.48:4303 , piece_id=WVHYGAZ6GHV2P7NRITNGHDWWEUKV3M3NMORMN52V5YKEGU3QHPJA, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.358575557 (381h42m12.229658602s) [noise=true , address=145.79.227.25:4617 , piece_id=SGHIRIQNT2ESPG2TBWSUFXAL4MVHI67NCDSJOEL7QAOGQYYFSB4A, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.360400502 (381h42m12.231483547s) [noise=true , address=145.79.227.45:4231 , piece_id=YKLGXTWGMIDYHHXSI6SJWNWWANNLMI5IDS5ETBNXKYZQ5ULZJ3BA, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.393842090 (381h42m12.264925145s) [noise=true , address=145.79.227.36:4517 , piece_id=PKQPPU7ZZE4NHX6RV6NXMOVWCOUMV5XDFOCJ4K7OSY6IFGTJMZFQ, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.399114879 (381h42m12.270197924s) [noise=true , address=145.79.227.15:4875 , piece_id=32EQTBBSDBPSJONWUXUT5GJTWPGT7ZKU7QJSJ4XGZBSDGTAIZFWA, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.421316085 (381h42m12.29239914s) [noise=true , address=145.79.227.42:4121 , piece_id=JDDCHIOAV2CZSSAEQAH3D5PCFFE2XQMLTJDOAZEFXIMQRVYA3NMA, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.562151156 (381h42m12.433234201s) [noise=true , address=77.90.31.12:4590 , piece_id=RXNC4MPRT7UBD27PM2AVGRC2HUJW3IQ5DKMYYPPFYEH44ZUTYUTA, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.576121679 (381h42m12.447204714s) [noise=true , address=45.91.116.9:4261 , piece_id=FY4CVGTSSU2KIFZTKXQLAZXAMZICM2K344TNQLJEES253AHRFC6A, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.622137751 (381h42m12.493220786s) [noise=true , address=77.90.31.14:4663 , piece_id=MMTEPTB6J3NWMT4T3IHEAJAG5YFFPTI56ZVW2PLGQWQWXLW5VJGQ, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.622162678 (381h42m12.493245724s) [seg_pos={0 0}, upload=334980]: Segment reached optimal threshold of 13 pieces. Dec 21 13:11:12.622427592 (381h42m12.493510647s) [noise=true , address=45.91.116.13:4154 , piece_id=3H4LTV35TLTMZCRXSSR4MQZ36QT5B7Z4WE2MGNJLIX77KTMZ4L3A, seg_pos={0 0}, upload=334980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 21 13:11:12.622436600 (381h42m12.493519635s) [noise=true , address=45.91.116.13:4154 , piece_id=3H4LTV35TLTMZCRXSSR4MQZ36QT5B7Z4WE2MGNJLIX77KTMZ4L3A, seg_pos={0 0}, upload=334980]: Long tail cancellation due to optimal threshold Dec 21 13:11:12.622546820 (381h42m12.493629855s) [noise=true , address=77.90.31.10:4517 , piece_id=HDGCMARA4JQEL4M45A3ASVLT3EFKBTG4OKV3YM2GPS7JHJCESW3Q, seg_pos={0 0}, upload=334980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 21 13:11:12.622564834 (381h42m12.493647879s) [noise=true , address=77.90.31.10:4517 , piece_id=HDGCMARA4JQEL4M45A3ASVLT3EFKBTG4OKV3YM2GPS7JHJCESW3Q, seg_pos={0 0}, upload=334980]: Long tail cancellation due to optimal threshold Dec 21 13:11:12.622606123 (381h42m12.493689158s) [noise=true , address=45.91.116.10:4082 , piece_id=LU2I6MA2UHRK44YDFMJKHJXJ5PUEX2CCCGDUYPKOTUCVV4TDSAOA, seg_pos={0 0}, upload=334980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 21 13:11:12.622615641 (381h42m12.493698676s) [noise=true , address=45.91.116.10:4082 , piece_id=LU2I6MA2UHRK44YDFMJKHJXJ5PUEX2CCCGDUYPKOTUCVV4TDSAOA, seg_pos={0 0}, upload=334980]: Long tail cancellation due to optimal threshold Dec 21 13:11:12.622639717 (381h42m12.493722752s) [seg_pos={0 0}, upload=334980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 13:11:12.622727554 (381h42m12.493810589s) [upload=334980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 13:11:12.622734707 (381h42m12.493817752s) [upload=334980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 13:11:12.934232308 (381h42m12.805315363s) [upload=334980]: Done uploading segments... Dec 21 13:11:12.934255752 (381h42m12.805338797s) [upload=334980]: Upload finished. err: <nil> Duration: 5.700372404s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X.....X..X.....X..X....X........X.....................XXXX..] duration: 5.949737903s trace: https://jaeger.dev.storj.io/trace/2ecb840715b827b4 FAIL (2025-12-21 13:12:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: FY4CVGTSSU2KIFZTKXQLAZXAMZICM2K344TNQLJEES253AHRFC6A) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: MMTEPTB6J3NWMT4T3IHEAJAG5YFFPTI56ZVW2PLGQWQWXLW5VJGQ) context deadline exceeded ======LOGS====== Dec 21 13:11:10.074980265 (381h42m9.94606331s) [upload=334980]: Starting upload Dec 21 13:11:10.075064275 (381h42m9.94614732s) [upload=334980]: Done starting upload Dec 21 13:11:10.075209582 (381h42m9.946292637s) [upload=334980]: Uploading segments... Dec 21 13:11:10.075280346 (381h42m9.946363392s) [upload=334980]: Got next segment. Inline: false Dec 21 13:11:10.075315323 (381h42m9.946398368s) [upload=334980]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 13:11:10.075320783 (381h42m9.946403828s) [upload=334980]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 13:11:10.327558205 (381h42m10.19864126s) [seg_pos={0 0}, upload=334980]: Begin upload segment... Dec 21 13:11:10.327625774 (381h42m10.198708809s) [seg_pos={0 0}, upload=334980]: Done begin upload segment. Dec 21 13:11:10.327682642 (381h42m10.198765677s) [noise=true , address=45.91.116.13:4154 , piece_id=3H4LTV35TLTMZCRXSSR4MQZ36QT5B7Z4WE2MGNJLIX77KTMZ4L3A, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.327709283 (381h42m10.198792328s) [noise=true , address=145.79.227.15:4875 , piece_id=32EQTBBSDBPSJONWUXUT5GJTWPGT7ZKU7QJSJ4XGZBSDGTAIZFWA, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.328281392 (381h42m10.199364467s) [noise=true , address=145.79.227.10:4403 , piece_id=O36LV7JYDF52VNSBPPG6XRHPJ4NKQIPBDYNHVKKWGMX2RBWM325A, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.328785282 (381h42m10.199868317s) [noise=true , address=145.79.227.32:4834 , piece_id=DUT5AUZUUETGQ57XEP25MYFD5HNYBYIWG6WFWXEE7A3NKKGEQ3OQ, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.328808836 (381h42m10.199891891s) [noise=true , address=45.91.116.9:4261 , piece_id=FY4CVGTSSU2KIFZTKXQLAZXAMZICM2K344TNQLJEES253AHRFC6A, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.329264193 (381h42m10.200347228s) [noise=true , address=77.90.31.14:4663 , piece_id=MMTEPTB6J3NWMT4T3IHEAJAG5YFFPTI56ZVW2PLGQWQWXLW5VJGQ, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.329720652 (381h42m10.200803698s) [noise=true , address=145.79.227.42:4121 , piece_id=JDDCHIOAV2CZSSAEQAH3D5PCFFE2XQMLTJDOAZEFXIMQRVYA3NMA, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.330129040 (381h42m10.201212075s) [noise=true , address=145.79.227.51:4411 , piece_id=KDCIGUCYVQXFJ7EEKKLYZYMDCTQLNOO7CPX7B5QUS2QC5NUNJJBQ, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.330507000 (381h42m10.201590035s) [noise=true , address=45.91.116.10:4082 , piece_id=LU2I6MA2UHRK44YDFMJKHJXJ5PUEX2CCCGDUYPKOTUCVV4TDSAOA, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.331607545 (381h42m10.20269058s) [noise=true , address=77.90.31.12:4590 , piece_id=RXNC4MPRT7UBD27PM2AVGRC2HUJW3IQ5DKMYYPPFYEH44ZUTYUTA, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.332067642 (381h42m10.203150677s) [noise=true , address=145.79.227.36:4517 , piece_id=PKQPPU7ZZE4NHX6RV6NXMOVWCOUMV5XDFOCJ4K7OSY6IFGTJMZFQ, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.333358941 (381h42m10.204441976s) [noise=true , address=77.90.31.10:4517 , piece_id=HDGCMARA4JQEL4M45A3ASVLT3EFKBTG4OKV3YM2GPS7JHJCESW3Q, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.336279552 (381h42m10.207362597s) [noise=true , address=145.79.227.37:4566 , piece_id=M5RUBUOWCO66UXO5BNCQT3I645D5NNUTYQPUD2Q57UO756ZSVIMA, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.342865683 (381h42m10.213948728s) [noise=true , address=145.79.227.48:4303 , piece_id=WVHYGAZ6GHV2P7NRITNGHDWWEUKV3M3NMORMN52V5YKEGU3QHPJA, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.345228553 (381h42m10.216311598s) [noise=true , address=145.79.227.25:4617 , piece_id=SGHIRIQNT2ESPG2TBWSUFXAL4MVHI67NCDSJOEL7QAOGQYYFSB4A, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.345748573 (381h42m10.216831618s) [noise=true , address=145.79.227.45:4231 , piece_id=YKLGXTWGMIDYHHXSI6SJWNWWANNLMI5IDS5ETBNXKYZQ5ULZJ3BA, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:12.050187306 (381h42m11.921270351s) [upload=334980]: Next returned nil segment Dec 21 13:11:12.050274632 (381h42m11.921357697s) [upload=334980]: Waiting for error group managing segments... Dec 21 13:11:12.333470151 (381h42m12.204553196s) [noise=true , address=145.79.227.51:4411 , piece_id=KDCIGUCYVQXFJ7EEKKLYZYMDCTQLNOO7CPX7B5QUS2QC5NUNJJBQ, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.334377990 (381h42m12.205461025s) [noise=true , address=145.79.227.32:4834 , piece_id=DUT5AUZUUETGQ57XEP25MYFD5HNYBYIWG6WFWXEE7A3NKKGEQ3OQ, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.338961147 (381h42m12.210044192s) [noise=true , address=145.79.227.10:4403 , piece_id=O36LV7JYDF52VNSBPPG6XRHPJ4NKQIPBDYNHVKKWGMX2RBWM325A, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.349152604 (381h42m12.220235649s) [noise=true , address=145.79.227.37:4566 , piece_id=M5RUBUOWCO66UXO5BNCQT3I645D5NNUTYQPUD2Q57UO756ZSVIMA, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.351128877 (381h42m12.222211922s) [noise=true , address=145.79.227.48:4303 , piece_id=WVHYGAZ6GHV2P7NRITNGHDWWEUKV3M3NMORMN52V5YKEGU3QHPJA, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.358575557 (381h42m12.229658602s) [noise=true , address=145.79.227.25:4617 , piece_id=SGHIRIQNT2ESPG2TBWSUFXAL4MVHI67NCDSJOEL7QAOGQYYFSB4A, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.360400502 (381h42m12.231483547s) [noise=true , address=145.79.227.45:4231 , piece_id=YKLGXTWGMIDYHHXSI6SJWNWWANNLMI5IDS5ETBNXKYZQ5ULZJ3BA, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.393842090 (381h42m12.264925145s) [noise=true , address=145.79.227.36:4517 , piece_id=PKQPPU7ZZE4NHX6RV6NXMOVWCOUMV5XDFOCJ4K7OSY6IFGTJMZFQ, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.399114879 (381h42m12.270197924s) [noise=true , address=145.79.227.15:4875 , piece_id=32EQTBBSDBPSJONWUXUT5GJTWPGT7ZKU7QJSJ4XGZBSDGTAIZFWA, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.421316085 (381h42m12.29239914s) [noise=true , address=145.79.227.42:4121 , piece_id=JDDCHIOAV2CZSSAEQAH3D5PCFFE2XQMLTJDOAZEFXIMQRVYA3NMA, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.562151156 (381h42m12.433234201s) [noise=true , address=77.90.31.12:4590 , piece_id=RXNC4MPRT7UBD27PM2AVGRC2HUJW3IQ5DKMYYPPFYEH44ZUTYUTA, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.576121679 (381h42m12.447204714s) [noise=true , address=45.91.116.9:4261 , piece_id=FY4CVGTSSU2KIFZTKXQLAZXAMZICM2K344TNQLJEES253AHRFC6A, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.622137751 (381h42m12.493220786s) [noise=true , address=77.90.31.14:4663 , piece_id=MMTEPTB6J3NWMT4T3IHEAJAG5YFFPTI56ZVW2PLGQWQWXLW5VJGQ, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.622162678 (381h42m12.493245724s) [seg_pos={0 0}, upload=334980]: Segment reached optimal threshold of 13 pieces. Dec 21 13:11:12.622427592 (381h42m12.493510647s) [noise=true , address=45.91.116.13:4154 , piece_id=3H4LTV35TLTMZCRXSSR4MQZ36QT5B7Z4WE2MGNJLIX77KTMZ4L3A, seg_pos={0 0}, upload=334980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 21 13:11:12.622436600 (381h42m12.493519635s) [noise=true , address=45.91.116.13:4154 , piece_id=3H4LTV35TLTMZCRXSSR4MQZ36QT5B7Z4WE2MGNJLIX77KTMZ4L3A, seg_pos={0 0}, upload=334980]: Long tail cancellation due to optimal threshold Dec 21 13:11:12.622546820 (381h42m12.493629855s) [noise=true , address=77.90.31.10:4517 , piece_id=HDGCMARA4JQEL4M45A3ASVLT3EFKBTG4OKV3YM2GPS7JHJCESW3Q, seg_pos={0 0}, upload=334980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 21 13:11:12.622564834 (381h42m12.493647879s) [noise=true , address=77.90.31.10:4517 , piece_id=HDGCMARA4JQEL4M45A3ASVLT3EFKBTG4OKV3YM2GPS7JHJCESW3Q, seg_pos={0 0}, upload=334980]: Long tail cancellation due to optimal threshold Dec 21 13:11:12.622606123 (381h42m12.493689158s) [noise=true , address=45.91.116.10:4082 , piece_id=LU2I6MA2UHRK44YDFMJKHJXJ5PUEX2CCCGDUYPKOTUCVV4TDSAOA, seg_pos={0 0}, upload=334980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 21 13:11:12.622615641 (381h42m12.493698676s) [noise=true , address=45.91.116.10:4082 , piece_id=LU2I6MA2UHRK44YDFMJKHJXJ5PUEX2CCCGDUYPKOTUCVV4TDSAOA, seg_pos={0 0}, upload=334980]: Long tail cancellation due to optimal threshold Dec 21 13:11:12.622639717 (381h42m12.493722752s) [seg_pos={0 0}, upload=334980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 13:11:12.622727554 (381h42m12.493810589s) [upload=334980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 13:11:12.622734707 (381h42m12.493817752s) [upload=334980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 13:11:12.934232308 (381h42m12.805315363s) [upload=334980]: Done uploading segments... Dec 21 13:11:12.934255752 (381h42m12.805338797s) [upload=334980]: Upload finished. err: <nil> Duration: 5.949819057s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X.....X..X.....X..X....X........X.....................XXXX.] duration: 5.656582208s trace: https://jaeger.dev.storj.io/trace/2ecb840715b827b4 FAIL (2025-12-21 13:11:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: FY4CVGTSSU2KIFZTKXQLAZXAMZICM2K344TNQLJEES253AHRFC6A) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: MMTEPTB6J3NWMT4T3IHEAJAG5YFFPTI56ZVW2PLGQWQWXLW5VJGQ) context deadline exceeded ======LOGS====== Dec 21 13:11:10.074980265 (381h42m9.94606331s) [upload=334980]: Starting upload Dec 21 13:11:10.075064275 (381h42m9.94614732s) [upload=334980]: Done starting upload Dec 21 13:11:10.075209582 (381h42m9.946292637s) [upload=334980]: Uploading segments... Dec 21 13:11:10.075280346 (381h42m9.946363392s) [upload=334980]: Got next segment. Inline: false Dec 21 13:11:10.075315323 (381h42m9.946398368s) [upload=334980]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 13:11:10.075320783 (381h42m9.946403828s) [upload=334980]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 13:11:10.327558205 (381h42m10.19864126s) [seg_pos={0 0}, upload=334980]: Begin upload segment... Dec 21 13:11:10.327625774 (381h42m10.198708809s) [seg_pos={0 0}, upload=334980]: Done begin upload segment. Dec 21 13:11:10.327682642 (381h42m10.198765677s) [noise=true , address=45.91.116.13:4154 , piece_id=3H4LTV35TLTMZCRXSSR4MQZ36QT5B7Z4WE2MGNJLIX77KTMZ4L3A, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.327709283 (381h42m10.198792328s) [noise=true , address=145.79.227.15:4875 , piece_id=32EQTBBSDBPSJONWUXUT5GJTWPGT7ZKU7QJSJ4XGZBSDGTAIZFWA, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.328281392 (381h42m10.199364467s) [noise=true , address=145.79.227.10:4403 , piece_id=O36LV7JYDF52VNSBPPG6XRHPJ4NKQIPBDYNHVKKWGMX2RBWM325A, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.328785282 (381h42m10.199868317s) [noise=true , address=145.79.227.32:4834 , piece_id=DUT5AUZUUETGQ57XEP25MYFD5HNYBYIWG6WFWXEE7A3NKKGEQ3OQ, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.328808836 (381h42m10.199891891s) [noise=true , address=45.91.116.9:4261 , piece_id=FY4CVGTSSU2KIFZTKXQLAZXAMZICM2K344TNQLJEES253AHRFC6A, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.329264193 (381h42m10.200347228s) [noise=true , address=77.90.31.14:4663 , piece_id=MMTEPTB6J3NWMT4T3IHEAJAG5YFFPTI56ZVW2PLGQWQWXLW5VJGQ, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.329720652 (381h42m10.200803698s) [noise=true , address=145.79.227.42:4121 , piece_id=JDDCHIOAV2CZSSAEQAH3D5PCFFE2XQMLTJDOAZEFXIMQRVYA3NMA, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.330129040 (381h42m10.201212075s) [noise=true , address=145.79.227.51:4411 , piece_id=KDCIGUCYVQXFJ7EEKKLYZYMDCTQLNOO7CPX7B5QUS2QC5NUNJJBQ, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.330507000 (381h42m10.201590035s) [noise=true , address=45.91.116.10:4082 , piece_id=LU2I6MA2UHRK44YDFMJKHJXJ5PUEX2CCCGDUYPKOTUCVV4TDSAOA, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.331607545 (381h42m10.20269058s) [noise=true , address=77.90.31.12:4590 , piece_id=RXNC4MPRT7UBD27PM2AVGRC2HUJW3IQ5DKMYYPPFYEH44ZUTYUTA, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.332067642 (381h42m10.203150677s) [noise=true , address=145.79.227.36:4517 , piece_id=PKQPPU7ZZE4NHX6RV6NXMOVWCOUMV5XDFOCJ4K7OSY6IFGTJMZFQ, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.333358941 (381h42m10.204441976s) [noise=true , address=77.90.31.10:4517 , piece_id=HDGCMARA4JQEL4M45A3ASVLT3EFKBTG4OKV3YM2GPS7JHJCESW3Q, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.336279552 (381h42m10.207362597s) [noise=true , address=145.79.227.37:4566 , piece_id=M5RUBUOWCO66UXO5BNCQT3I645D5NNUTYQPUD2Q57UO756ZSVIMA, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.342865683 (381h42m10.213948728s) [noise=true , address=145.79.227.48:4303 , piece_id=WVHYGAZ6GHV2P7NRITNGHDWWEUKV3M3NMORMN52V5YKEGU3QHPJA, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.345228553 (381h42m10.216311598s) [noise=true , address=145.79.227.25:4617 , piece_id=SGHIRIQNT2ESPG2TBWSUFXAL4MVHI67NCDSJOEL7QAOGQYYFSB4A, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.345748573 (381h42m10.216831618s) [noise=true , address=145.79.227.45:4231 , piece_id=YKLGXTWGMIDYHHXSI6SJWNWWANNLMI5IDS5ETBNXKYZQ5ULZJ3BA, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:12.050187306 (381h42m11.921270351s) [upload=334980]: Next returned nil segment Dec 21 13:11:12.050274632 (381h42m11.921357697s) [upload=334980]: Waiting for error group managing segments... Dec 21 13:11:12.333470151 (381h42m12.204553196s) [noise=true , address=145.79.227.51:4411 , piece_id=KDCIGUCYVQXFJ7EEKKLYZYMDCTQLNOO7CPX7B5QUS2QC5NUNJJBQ, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.334377990 (381h42m12.205461025s) [noise=true , address=145.79.227.32:4834 , piece_id=DUT5AUZUUETGQ57XEP25MYFD5HNYBYIWG6WFWXEE7A3NKKGEQ3OQ, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.338961147 (381h42m12.210044192s) [noise=true , address=145.79.227.10:4403 , piece_id=O36LV7JYDF52VNSBPPG6XRHPJ4NKQIPBDYNHVKKWGMX2RBWM325A, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.349152604 (381h42m12.220235649s) [noise=true , address=145.79.227.37:4566 , piece_id=M5RUBUOWCO66UXO5BNCQT3I645D5NNUTYQPUD2Q57UO756ZSVIMA, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.351128877 (381h42m12.222211922s) [noise=true , address=145.79.227.48:4303 , piece_id=WVHYGAZ6GHV2P7NRITNGHDWWEUKV3M3NMORMN52V5YKEGU3QHPJA, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.358575557 (381h42m12.229658602s) [noise=true , address=145.79.227.25:4617 , piece_id=SGHIRIQNT2ESPG2TBWSUFXAL4MVHI67NCDSJOEL7QAOGQYYFSB4A, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.360400502 (381h42m12.231483547s) [noise=true , address=145.79.227.45:4231 , piece_id=YKLGXTWGMIDYHHXSI6SJWNWWANNLMI5IDS5ETBNXKYZQ5ULZJ3BA, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.393842090 (381h42m12.264925145s) [noise=true , address=145.79.227.36:4517 , piece_id=PKQPPU7ZZE4NHX6RV6NXMOVWCOUMV5XDFOCJ4K7OSY6IFGTJMZFQ, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.399114879 (381h42m12.270197924s) [noise=true , address=145.79.227.15:4875 , piece_id=32EQTBBSDBPSJONWUXUT5GJTWPGT7ZKU7QJSJ4XGZBSDGTAIZFWA, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.421316085 (381h42m12.29239914s) [noise=true , address=145.79.227.42:4121 , piece_id=JDDCHIOAV2CZSSAEQAH3D5PCFFE2XQMLTJDOAZEFXIMQRVYA3NMA, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.562151156 (381h42m12.433234201s) [noise=true , address=77.90.31.12:4590 , piece_id=RXNC4MPRT7UBD27PM2AVGRC2HUJW3IQ5DKMYYPPFYEH44ZUTYUTA, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.576121679 (381h42m12.447204714s) [noise=true , address=45.91.116.9:4261 , piece_id=FY4CVGTSSU2KIFZTKXQLAZXAMZICM2K344TNQLJEES253AHRFC6A, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.622137751 (381h42m12.493220786s) [noise=true , address=77.90.31.14:4663 , piece_id=MMTEPTB6J3NWMT4T3IHEAJAG5YFFPTI56ZVW2PLGQWQWXLW5VJGQ, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.622162678 (381h42m12.493245724s) [seg_pos={0 0}, upload=334980]: Segment reached optimal threshold of 13 pieces. Dec 21 13:11:12.622427592 (381h42m12.493510647s) [noise=true , address=45.91.116.13:4154 , piece_id=3H4LTV35TLTMZCRXSSR4MQZ36QT5B7Z4WE2MGNJLIX77KTMZ4L3A, seg_pos={0 0}, upload=334980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 21 13:11:12.622436600 (381h42m12.493519635s) [noise=true , address=45.91.116.13:4154 , piece_id=3H4LTV35TLTMZCRXSSR4MQZ36QT5B7Z4WE2MGNJLIX77KTMZ4L3A, seg_pos={0 0}, upload=334980]: Long tail cancellation due to optimal threshold Dec 21 13:11:12.622546820 (381h42m12.493629855s) [noise=true , address=77.90.31.10:4517 , piece_id=HDGCMARA4JQEL4M45A3ASVLT3EFKBTG4OKV3YM2GPS7JHJCESW3Q, seg_pos={0 0}, upload=334980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 21 13:11:12.622564834 (381h42m12.493647879s) [noise=true , address=77.90.31.10:4517 , piece_id=HDGCMARA4JQEL4M45A3ASVLT3EFKBTG4OKV3YM2GPS7JHJCESW3Q, seg_pos={0 0}, upload=334980]: Long tail cancellation due to optimal threshold Dec 21 13:11:12.622606123 (381h42m12.493689158s) [noise=true , address=45.91.116.10:4082 , piece_id=LU2I6MA2UHRK44YDFMJKHJXJ5PUEX2CCCGDUYPKOTUCVV4TDSAOA, seg_pos={0 0}, upload=334980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 21 13:11:12.622615641 (381h42m12.493698676s) [noise=true , address=45.91.116.10:4082 , piece_id=LU2I6MA2UHRK44YDFMJKHJXJ5PUEX2CCCGDUYPKOTUCVV4TDSAOA, seg_pos={0 0}, upload=334980]: Long tail cancellation due to optimal threshold Dec 21 13:11:12.622639717 (381h42m12.493722752s) [seg_pos={0 0}, upload=334980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 13:11:12.622727554 (381h42m12.493810589s) [upload=334980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 13:11:12.622734707 (381h42m12.493817752s) [upload=334980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 13:11:12.934232308 (381h42m12.805315363s) [upload=334980]: Done uploading segments... Dec 21 13:11:12.934255752 (381h42m12.805338797s) [upload=334980]: Upload finished. err: <nil> Duration: 5.656600203s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X.X.....X..X.....X..X....X........X.....................XXXX] duration: 30.003144737s trace: https://jaeger.dev.storj.io/trace/2ecb840715b827b4 FAIL (2025-12-21 13:11:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: FY4CVGTSSU2KIFZTKXQLAZXAMZICM2K344TNQLJEES253AHRFC6A) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: MMTEPTB6J3NWMT4T3IHEAJAG5YFFPTI56ZVW2PLGQWQWXLW5VJGQ) context deadline exceeded ======LOGS====== Dec 21 13:11:10.074980265 (381h42m9.94606331s) [upload=334980]: Starting upload Dec 21 13:11:10.075064275 (381h42m9.94614732s) [upload=334980]: Done starting upload Dec 21 13:11:10.075209582 (381h42m9.946292637s) [upload=334980]: Uploading segments... Dec 21 13:11:10.075280346 (381h42m9.946363392s) [upload=334980]: Got next segment. Inline: false Dec 21 13:11:10.075315323 (381h42m9.946398368s) [upload=334980]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 13:11:10.075320783 (381h42m9.946403828s) [upload=334980]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 13:11:10.327558205 (381h42m10.19864126s) [seg_pos={0 0}, upload=334980]: Begin upload segment... Dec 21 13:11:10.327625774 (381h42m10.198708809s) [seg_pos={0 0}, upload=334980]: Done begin upload segment. Dec 21 13:11:10.327682642 (381h42m10.198765677s) [noise=true , address=45.91.116.13:4154 , piece_id=3H4LTV35TLTMZCRXSSR4MQZ36QT5B7Z4WE2MGNJLIX77KTMZ4L3A, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.327709283 (381h42m10.198792328s) [noise=true , address=145.79.227.15:4875 , piece_id=32EQTBBSDBPSJONWUXUT5GJTWPGT7ZKU7QJSJ4XGZBSDGTAIZFWA, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.328281392 (381h42m10.199364467s) [noise=true , address=145.79.227.10:4403 , piece_id=O36LV7JYDF52VNSBPPG6XRHPJ4NKQIPBDYNHVKKWGMX2RBWM325A, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.328785282 (381h42m10.199868317s) [noise=true , address=145.79.227.32:4834 , piece_id=DUT5AUZUUETGQ57XEP25MYFD5HNYBYIWG6WFWXEE7A3NKKGEQ3OQ, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.328808836 (381h42m10.199891891s) [noise=true , address=45.91.116.9:4261 , piece_id=FY4CVGTSSU2KIFZTKXQLAZXAMZICM2K344TNQLJEES253AHRFC6A, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.329264193 (381h42m10.200347228s) [noise=true , address=77.90.31.14:4663 , piece_id=MMTEPTB6J3NWMT4T3IHEAJAG5YFFPTI56ZVW2PLGQWQWXLW5VJGQ, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.329720652 (381h42m10.200803698s) [noise=true , address=145.79.227.42:4121 , piece_id=JDDCHIOAV2CZSSAEQAH3D5PCFFE2XQMLTJDOAZEFXIMQRVYA3NMA, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.330129040 (381h42m10.201212075s) [noise=true , address=145.79.227.51:4411 , piece_id=KDCIGUCYVQXFJ7EEKKLYZYMDCTQLNOO7CPX7B5QUS2QC5NUNJJBQ, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.330507000 (381h42m10.201590035s) [noise=true , address=45.91.116.10:4082 , piece_id=LU2I6MA2UHRK44YDFMJKHJXJ5PUEX2CCCGDUYPKOTUCVV4TDSAOA, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.331607545 (381h42m10.20269058s) [noise=true , address=77.90.31.12:4590 , piece_id=RXNC4MPRT7UBD27PM2AVGRC2HUJW3IQ5DKMYYPPFYEH44ZUTYUTA, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.332067642 (381h42m10.203150677s) [noise=true , address=145.79.227.36:4517 , piece_id=PKQPPU7ZZE4NHX6RV6NXMOVWCOUMV5XDFOCJ4K7OSY6IFGTJMZFQ, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.333358941 (381h42m10.204441976s) [noise=true , address=77.90.31.10:4517 , piece_id=HDGCMARA4JQEL4M45A3ASVLT3EFKBTG4OKV3YM2GPS7JHJCESW3Q, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.336279552 (381h42m10.207362597s) [noise=true , address=145.79.227.37:4566 , piece_id=M5RUBUOWCO66UXO5BNCQT3I645D5NNUTYQPUD2Q57UO756ZSVIMA, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.342865683 (381h42m10.213948728s) [noise=true , address=145.79.227.48:4303 , piece_id=WVHYGAZ6GHV2P7NRITNGHDWWEUKV3M3NMORMN52V5YKEGU3QHPJA, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.345228553 (381h42m10.216311598s) [noise=true , address=145.79.227.25:4617 , piece_id=SGHIRIQNT2ESPG2TBWSUFXAL4MVHI67NCDSJOEL7QAOGQYYFSB4A, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:10.345748573 (381h42m10.216831618s) [noise=true , address=145.79.227.45:4231 , piece_id=YKLGXTWGMIDYHHXSI6SJWNWWANNLMI5IDS5ETBNXKYZQ5ULZJ3BA, seg_pos={0 0}, upload=334980]: Uploading piece... Dec 21 13:11:12.050187306 (381h42m11.921270351s) [upload=334980]: Next returned nil segment Dec 21 13:11:12.050274632 (381h42m11.921357697s) [upload=334980]: Waiting for error group managing segments... Dec 21 13:11:12.333470151 (381h42m12.204553196s) [noise=true , address=145.79.227.51:4411 , piece_id=KDCIGUCYVQXFJ7EEKKLYZYMDCTQLNOO7CPX7B5QUS2QC5NUNJJBQ, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.334377990 (381h42m12.205461025s) [noise=true , address=145.79.227.32:4834 , piece_id=DUT5AUZUUETGQ57XEP25MYFD5HNYBYIWG6WFWXEE7A3NKKGEQ3OQ, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.338961147 (381h42m12.210044192s) [noise=true , address=145.79.227.10:4403 , piece_id=O36LV7JYDF52VNSBPPG6XRHPJ4NKQIPBDYNHVKKWGMX2RBWM325A, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.349152604 (381h42m12.220235649s) [noise=true , address=145.79.227.37:4566 , piece_id=M5RUBUOWCO66UXO5BNCQT3I645D5NNUTYQPUD2Q57UO756ZSVIMA, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.351128877 (381h42m12.222211922s) [noise=true , address=145.79.227.48:4303 , piece_id=WVHYGAZ6GHV2P7NRITNGHDWWEUKV3M3NMORMN52V5YKEGU3QHPJA, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.358575557 (381h42m12.229658602s) [noise=true , address=145.79.227.25:4617 , piece_id=SGHIRIQNT2ESPG2TBWSUFXAL4MVHI67NCDSJOEL7QAOGQYYFSB4A, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.360400502 (381h42m12.231483547s) [noise=true , address=145.79.227.45:4231 , piece_id=YKLGXTWGMIDYHHXSI6SJWNWWANNLMI5IDS5ETBNXKYZQ5ULZJ3BA, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.393842090 (381h42m12.264925145s) [noise=true , address=145.79.227.36:4517 , piece_id=PKQPPU7ZZE4NHX6RV6NXMOVWCOUMV5XDFOCJ4K7OSY6IFGTJMZFQ, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.399114879 (381h42m12.270197924s) [noise=true , address=145.79.227.15:4875 , piece_id=32EQTBBSDBPSJONWUXUT5GJTWPGT7ZKU7QJSJ4XGZBSDGTAIZFWA, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.421316085 (381h42m12.29239914s) [noise=true , address=145.79.227.42:4121 , piece_id=JDDCHIOAV2CZSSAEQAH3D5PCFFE2XQMLTJDOAZEFXIMQRVYA3NMA, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.562151156 (381h42m12.433234201s) [noise=true , address=77.90.31.12:4590 , piece_id=RXNC4MPRT7UBD27PM2AVGRC2HUJW3IQ5DKMYYPPFYEH44ZUTYUTA, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.576121679 (381h42m12.447204714s) [noise=true , address=45.91.116.9:4261 , piece_id=FY4CVGTSSU2KIFZTKXQLAZXAMZICM2K344TNQLJEES253AHRFC6A, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.622137751 (381h42m12.493220786s) [noise=true , address=77.90.31.14:4663 , piece_id=MMTEPTB6J3NWMT4T3IHEAJAG5YFFPTI56ZVW2PLGQWQWXLW5VJGQ, seg_pos={0 0}, upload=334980]: Done uploading piece. err: <nil> Dec 21 13:11:12.622162678 (381h42m12.493245724s) [seg_pos={0 0}, upload=334980]: Segment reached optimal threshold of 13 pieces. Dec 21 13:11:12.622427592 (381h42m12.493510647s) [noise=true , address=45.91.116.13:4154 , piece_id=3H4LTV35TLTMZCRXSSR4MQZ36QT5B7Z4WE2MGNJLIX77KTMZ4L3A, seg_pos={0 0}, upload=334980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 21 13:11:12.622436600 (381h42m12.493519635s) [noise=true , address=45.91.116.13:4154 , piece_id=3H4LTV35TLTMZCRXSSR4MQZ36QT5B7Z4WE2MGNJLIX77KTMZ4L3A, seg_pos={0 0}, upload=334980]: Long tail cancellation due to optimal threshold Dec 21 13:11:12.622546820 (381h42m12.493629855s) [noise=true , address=77.90.31.10:4517 , piece_id=HDGCMARA4JQEL4M45A3ASVLT3EFKBTG4OKV3YM2GPS7JHJCESW3Q, seg_pos={0 0}, upload=334980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 21 13:11:12.622564834 (381h42m12.493647879s) [noise=true , address=77.90.31.10:4517 , piece_id=HDGCMARA4JQEL4M45A3ASVLT3EFKBTG4OKV3YM2GPS7JHJCESW3Q, seg_pos={0 0}, upload=334980]: Long tail cancellation due to optimal threshold Dec 21 13:11:12.622606123 (381h42m12.493689158s) [noise=true , address=45.91.116.10:4082 , piece_id=LU2I6MA2UHRK44YDFMJKHJXJ5PUEX2CCCGDUYPKOTUCVV4TDSAOA, seg_pos={0 0}, upload=334980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 21 13:11:12.622615641 (381h42m12.493698676s) [noise=true , address=45.91.116.10:4082 , piece_id=LU2I6MA2UHRK44YDFMJKHJXJ5PUEX2CCCGDUYPKOTUCVV4TDSAOA, seg_pos={0 0}, upload=334980]: Long tail cancellation due to optimal threshold Dec 21 13:11:12.622639717 (381h42m12.493722752s) [seg_pos={0 0}, upload=334980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 13:11:12.622727554 (381h42m12.493810589s) [upload=334980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 13:11:12.622734707 (381h42m12.493817752s) [upload=334980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 13:11:12.934232308 (381h42m12.805315363s) [upload=334980]: Done uploading segments... Dec 21 13:11:12.934255752 (381h42m12.805338797s) [upload=334980]: Upload finished. err: <nil> Duration: 30.003170726s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X.X.....X..X.....X..X....X........X.....................XXX] duration: 30.003414183s trace: https://jaeger.dev.storj.io/trace/1629b0bccf9365f1 FAIL (2025-12-21 13:10:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 13:09:56.347561627 (381h40m56.218644672s) [upload=334963]: Starting upload Dec 21 13:09:56.347624606 (381h40m56.218707641s) [upload=334963]: Done starting upload Dec 21 13:09:56.347669943 (381h40m56.218752998s) [upload=334963]: Uploading segments... Dec 21 13:09:56.347826361 (381h40m56.218909396s) [upload=334963]: Got next segment. Inline: false Dec 21 13:09:56.347865345 (381h40m56.21894838s) [upload=334963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 13:09:56.347869913 (381h40m56.218952959s) [upload=334963]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 13:09:56.593286696 (381h40m56.464369731s) [seg_pos={0 0}, upload=334963]: Begin upload segment... Dec 21 13:09:56.593338335 (381h40m56.46442137s) [seg_pos={0 0}, upload=334963]: Done begin upload segment. Dec 21 13:09:56.593415392 (381h40m56.464498437s) [noise=true , address=145.79.227.48:4302 , piece_id=TXISVRBIBAFZ5RZ63YP2UB7NM5P2AJZQK7ZN5A43AGXFVP723PSQ, seg_pos={0 0}, upload=334963]: Uploading piece... Dec 21 13:09:56.593436683 (381h40m56.464519718s) [noise=true , address=145.79.227.33:4871 , piece_id=4OBRS34D6VTSORWSLJG5EORNZNYEIFGWJ2FSDEC7R4JVVRYVRBHQ, seg_pos={0 0}, upload=334963]: Uploading piece... Dec 21 13:09:56.594139020 (381h40m56.465222055s) [noise=true , address=45.91.116.5:4300 , piece_id=43WPSQUC6KZMIH75SZITD6VQXHETAJNZ7AS4P5KC3KSJC3RLFKCA, seg_pos={0 0}, upload=334963]: Uploading piece... Dec 21 13:09:56.594584739 (381h40m56.465667784s) [noise=true , address=145.79.227.51:4410 , piece_id=5DROMBA35IM2WWVMAT7EUVN5CYMWZM5LLEWDACYGW6RH6CHMPRIA, seg_pos={0 0}, upload=334963]: Uploading piece... Dec 21 13:09:56.595014457 (381h40m56.466097492s) [noise=true , address=145.79.227.11:4731 , piece_id=ALKYBSYGFCSKPITWO4PJNWKJFOK2FVUOMAQAD74NZAYL6T42NW4A, seg_pos={0 0}, upload=334963]: Uploading piece... Dec 21 13:09:56.596205805 (381h40m56.46728885s) [noise=true , address=145.79.227.49:4337 , piece_id=MHCKROFFPVNMXBZPQEQ7TNYF5DYNKKRT4KJABDTX23H6PXWJCVUQ, seg_pos={0 0}, upload=334963]: Uploading piece... Dec 21 13:09:56.597496693 (381h40m56.468579728s) [noise=true , address=145.79.227.32:4835 , piece_id=U72RJAB4ONGNLG6XM4X66NMWXEGKB3IYKQA7OGMJRK24KSIZWX7Q, seg_pos={0 0}, upload=334963]: Uploading piece... Dec 21 13:09:56.598619131 (381h40m56.469702176s) [noise=true , address=145.79.227.10:4401 , piece_id=UTBQJ6PF7PRBKUFNCCLUCV46AIGLJTKER4CAB75J5TEA6SOKYRHA, seg_pos={0 0}, upload=334963]: Uploading piece... Dec 21 13:09:56.600832886 (381h40m56.471915931s) [noise=true , address=145.79.227.29:4943 , piece_id=GWZSHLNOLL7JR4UYRAO25TBKQ2G4AS5OYJRDZEMWUYMSXKVW2Z6Q, seg_pos={0 0}, upload=334963]: Uploading piece... Dec 21 13:09:56.603175386 (381h40m56.474258431s) [noise=true , address=145.79.227.45:4230 , piece_id=RSXLTCHCVUI3YDOBLSOEGHSDVMAVGJHRO2LO666FMAZWEBDUACXQ, seg_pos={0 0}, upload=334963]: Uploading piece... Dec 21 13:09:56.604108723 (381h40m56.475191759s) [noise=true , address=145.79.227.20:4437 , piece_id=PHN36IQE6TPIQZBNB7E6FSTSTASQVMI6G5MEMZ4DZKUKD7LE6OVQ, seg_pos={0 0}, upload=334963]: Uploading piece... Dec 21 13:09:56.605481338 (381h40m56.476564383s) [noise=true , address=45.91.116.12:4226 , piece_id=WXRUQIVRXGN7EKK7HU4X2H6SQG46EMZO5WOXFWNMGIXRKIIFWTLQ, seg_pos={0 0}, upload=334963]: Uploading piece... Dec 21 13:09:56.605794304 (381h40m56.476877349s) [noise=true , address=77.90.31.21:4877 , piece_id=4H26NK46WK5PP47QN553FVEEARAGH6HJMV5AX4TBUMRMBKQP7PMA, seg_pos={0 0}, upload=334963]: Uploading piece... Dec 21 13:09:56.607301994 (381h40m56.478385039s) [noise=true , address=145.79.227.35:4482 , piece_id=SL56SX32FNEQG3GVE7SM2YFOYG6V6E6ND6UVIGTK36C6BUU4TRHQ, seg_pos={0 0}, upload=334963]: Uploading piece... Dec 21 13:09:56.607452440 (381h40m56.478535475s) [noise=true , address=45.91.116.4:4940 , piece_id=PXAIZWMJQGTNRR2TGTXYPUHO5EGWX4HMZSPLNM7OSZZDHSVZIOUQ, seg_pos={0 0}, upload=334963]: Uploading piece... Dec 21 13:09:56.608975391 (381h40m56.480058426s) [noise=true , address=145.79.227.36:4519 , piece_id=YLTHJMNTBBBZCHAOX42Y73ATKKTEWUVQLYUT2CG62JQMSAYFR2NA, seg_pos={0 0}, upload=334963]: Uploading piece... Dec 21 13:09:57.718267331 (381h40m57.589350366s) [noise=true , address=77.90.31.21:4877 , piece_id=4H26NK46WK5PP47QN553FVEEARAGH6HJMV5AX4TBUMRMBKQP7PMA, seg_pos={0 0}, upload=334963]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: storage node overloaded, request limit: 2000 Dec 21 13:09:58.289783198 (381h40m58.160866243s) [upload=334963]: Next returned nil segment Dec 21 13:09:58.289817593 (381h40m58.160900638s) [upload=334963]: Waiting for error group managing segments... Dec 21 13:09:58.575228631 (381h40m58.446311676s) [noise=true , address=145.79.227.10:4401 , piece_id=UTBQJ6PF7PRBKUFNCCLUCV46AIGLJTKER4CAB75J5TEA6SOKYRHA, seg_pos={0 0}, upload=334963]: Done uploading piece. err: <nil> Dec 21 13:09:58.585192655 (381h40m58.45627569s) [noise=true , address=145.79.227.51:4410 , piece_id=5DROMBA35IM2WWVMAT7EUVN5CYMWZM5LLEWDACYGW6RH6CHMPRIA, seg_pos={0 0}, upload=334963]: Done uploading piece. err: <nil> Dec 21 13:09:58.586467031 (381h40m58.457550066s) [noise=true , address=145.79.227.48:4302 , piece_id=TXISVRBIBAFZ5RZ63YP2UB7NM5P2AJZQK7ZN5A43AGXFVP723PSQ, seg_pos={0 0}, upload=334963]: Done uploading piece. err: <nil> Dec 21 13:09:58.604728645 (381h40m58.47581169s) [noise=true , address=145.79.227.33:4871 , piece_id=4OBRS34D6VTSORWSLJG5EORNZNYEIFGWJ2FSDEC7R4JVVRYVRBHQ, seg_pos={0 0}, upload=334963]: Done uploading piece. err: <nil> Dec 21 13:09:58.615479165 (381h40m58.4865622s) [noise=true , address=145.79.227.32:4835 , piece_id=U72RJAB4ONGNLG6XM4X66NMWXEGKB3IYKQA7OGMJRK24KSIZWX7Q, seg_pos={0 0}, upload=334963]: Done uploading piece. err: <nil> Dec 21 13:09:58.615956975 (381h40m58.48704002s) [noise=true , address=145.79.227.20:4437 , piece_id=PHN36IQE6TPIQZBNB7E6FSTSTASQVMI6G5MEMZ4DZKUKD7LE6OVQ, seg_pos={0 0}, upload=334963]: Done uploading piece. err: <nil> Dec 21 13:09:58.616189107 (381h40m58.487272152s) [noise=true , address=145.79.227.29:4943 , piece_id=GWZSHLNOLL7JR4UYRAO25TBKQ2G4AS5OYJRDZEMWUYMSXKVW2Z6Q, seg_pos={0 0}, upload=334963]: Done uploading piece. err: <nil> Dec 21 13:09:58.618241455 (381h40m58.4893245s) [noise=true , address=145.79.227.45:4230 , piece_id=RSXLTCHCVUI3YDOBLSOEGHSDVMAVGJHRO2LO666FMAZWEBDUACXQ, seg_pos={0 0}, upload=334963]: Done uploading piece. err: <nil> Dec 21 13:09:58.618612491 (381h40m58.489695536s) [noise=true , address=145.79.227.36:4519 , piece_id=YLTHJMNTBBBZCHAOX42Y73ATKKTEWUVQLYUT2CG62JQMSAYFR2NA, seg_pos={0 0}, upload=334963]: Done uploading piece. err: <nil> Dec 21 13:09:58.624856431 (381h40m58.495939477s) [noise=true , address=145.79.227.35:4482 , piece_id=SL56SX32FNEQG3GVE7SM2YFOYG6V6E6ND6UVIGTK36C6BUU4TRHQ, seg_pos={0 0}, upload=334963]: Done uploading piece. err: <nil> Dec 21 13:09:58.651767183 (381h40m58.522850228s) [noise=true , address=145.79.227.11:4731 , piece_id=ALKYBSYGFCSKPITWO4PJNWKJFOK2FVUOMAQAD74NZAYL6T42NW4A, seg_pos={0 0}, upload=334963]: Done uploading piece. err: <nil> Dec 21 13:09:58.654468728 (381h40m58.525551763s) [noise=true , address=145.79.227.49:4337 , piece_id=MHCKROFFPVNMXBZPQEQ7TNYF5DYNKKRT4KJABDTX23H6PXWJCVUQ, seg_pos={0 0}, upload=334963]: Done uploading piece. err: <nil> Dec 21 13:09:58.840423675 (381h40m58.71150672s) [noise=true , address=45.91.116.5:4300 , piece_id=43WPSQUC6KZMIH75SZITD6VQXHETAJNZ7AS4P5KC3KSJC3RLFKCA, seg_pos={0 0}, upload=334963]: Done uploading piece. err: <nil> Dec 21 13:09:58.840448473 (381h40m58.711531518s) [seg_pos={0 0}, upload=334963]: Segment reached optimal threshold of 13 pieces. Dec 21 13:09:58.840608808 (381h40m58.711691853s) [noise=true , address=45.91.116.12:4226 , piece_id=WXRUQIVRXGN7EKK7HU4X2H6SQG46EMZO5WOXFWNMGIXRKIIFWTLQ, seg_pos={0 0}, upload=334963]: Done uploading piece. err: <nil> Dec 21 13:09:58.840830099 (381h40m58.711913144s) [noise=true , address=45.91.116.4:4940 , piece_id=PXAIZWMJQGTNRR2TGTXYPUHO5EGWX4HMZSPLNM7OSZZDHSVZIOUQ, seg_pos={0 0}, upload=334963]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 21 13:09:58.840840759 (381h40m58.711923794s) [noise=true , address=45.91.116.4:4940 , piece_id=PXAIZWMJQGTNRR2TGTXYPUHO5EGWX4HMZSPLNM7OSZZDHSVZIOUQ, seg_pos={0 0}, upload=334963]: Long tail cancellation due to optimal threshold Dec 21 13:09:58.840860217 (381h40m58.711943262s) [seg_pos={0 0}, upload=334963]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 21 13:09:58.841016504 (381h40m58.712099539s) [upload=334963]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 13:09:58.841025061 (381h40m58.712108106s) [upload=334963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 13:09:59.132426656 (381h40m59.003509691s) [upload=334963]: Done uploading segments... Dec 21 13:09:59.132445092 (381h40m59.003528127s) [upload=334963]: Upload finished. err: <nil> Duration: 30.003442036s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X.X.X.....X..X.....X..X....X........X.....................XX] duration: 30.004785897s trace: https://jaeger.dev.storj.io/trace/19031474d1ff1979 FAIL (2025-12-21 13:09:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, Piece ID: Y52FXYDIGRHET2SKURCZPVZ5AKQH6P62YCZ5RCPWR6MC2ZMM7DGQ) context deadline exceeded; piecestore: (Node ID: 1uzGTks5tffchQBGQK23fnWZXMfrsTHh4Zqs9UGwnWUmnQghEW, Piece ID: 7SSF7C7LW4AYMNTLXRCI77OAWULVIWUUTPER6XLURIMML3CDZVGQ) context deadline exceeded; piecestore: (Node ID: 1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE, Piece ID: DB3S7DUZSI5KIAKS7FQR6JHZEB3I7IDLBWEVJ2OHK7LOQ2OSSMXA) context deadline exceeded ======LOGS====== Dec 21 13:09:15.482318454 (381h40m15.353401499s) [upload=334951]: Starting upload Dec 21 13:09:15.482420618 (381h40m15.353503664s) [upload=334951]: Done starting upload Dec 21 13:09:15.482481433 (381h40m15.353564489s) [upload=334951]: Uploading segments... Dec 21 13:09:15.482733725 (381h40m15.35381677s) [upload=334951]: Got next segment. Inline: false Dec 21 13:09:15.482777418 (381h40m15.353860453s) [upload=334951]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 13:09:15.482783038 (381h40m15.353866083s) [upload=334951]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 13:09:15.724752938 (381h40m15.595835973s) [seg_pos={0 0}, upload=334951]: Begin upload segment... Dec 21 13:09:15.724803524 (381h40m15.59588656s) [seg_pos={0 0}, upload=334951]: Done begin upload segment. Dec 21 13:09:15.724831417 (381h40m15.595914462s) [noise=true , address=45.91.116.4:4939 , piece_id=7ROKKNPA22JAG3J4CMFI7WH2MB7DQFLPNL7GHZI4U27NO2RRP7DQ, seg_pos={0 0}, upload=334951]: Uploading piece... Dec 21 13:09:15.724845394 (381h40m15.595928459s) [noise=true , address=145.79.227.47:4266 , piece_id=7SSF7C7LW4AYMNTLXRCI77OAWULVIWUUTPER6XLURIMML3CDZVGQ, seg_pos={0 0}, upload=334951]: Uploading piece... Dec 21 13:09:15.725241358 (381h40m15.596324393s) [noise=true , address=145.79.227.46:4601 , piece_id=T5K4T3VUW3S43EKYTLBOU2ISNDXYMABN4O4IGYU2PDPOX47HKMEQ, seg_pos={0 0}, upload=334951]: Uploading piece... Dec 21 13:09:15.725510741 (381h40m15.596593776s) [noise=true , address=145.79.227.20:4439 , piece_id=DEV552BLOMSZGVWSY3WDS4IMQNJ3PU2K3BSBSRVZCIWPD5NGR6FQ, seg_pos={0 0}, upload=334951]: Uploading piece... Dec 21 13:09:15.726343577 (381h40m15.597426622s) [noise=true , address=145.79.227.16:4911 , piece_id=T3IVA3SR72JPXQERVZQPFHIAIHJDG6PIQHX2QPBY4LE7T2AI6EYA, seg_pos={0 0}, upload=334951]: Uploading piece... Dec 21 13:09:15.726649920 (381h40m15.597732955s) [noise=true , address=145.79.227.51:4411 , piece_id=EW5M23F4QLXZCVGVHHEVWH74BVXZNPS7JEBEAO6PJK7PEDTN4LXA, seg_pos={0 0}, upload=334951]: Uploading piece... Dec 21 13:09:15.726993164 (381h40m15.598076199s) [noise=true , address=145.79.227.37:4566 , piece_id=SNAWXEJ52STAJVCB6BXUXN4ETOIASJGIUS5IK4XEOZUYFDAXTJ3Q, seg_pos={0 0}, upload=334951]: Uploading piece... Dec 21 13:09:15.727306110 (381h40m15.598389145s) [noise=true , address=45.91.116.7:4012 , piece_id=OGMJVPHM74QXHBZLFRYWBNBTRBU5TYUINRP7MXBQMNT44AWUUEAA, seg_pos={0 0}, upload=334951]: Uploading piece... Dec 21 13:09:15.727726230 (381h40m15.598809265s) [noise=true , address=145.79.227.32:4834 , piece_id=P6OD6IOQL5BP7MHSNYJWET7NABCA2HVDW7WLI52TYTTAAM4FL3XA, seg_pos={0 0}, upload=334951]: Uploading piece... Dec 21 13:09:15.728652072 (381h40m15.599735107s) [noise=true , address=45.91.116.10:4084 , piece_id=C5WX3TLQJFMTXZGTGXY3MRCOIQCNA3S6JKNDNNYRR5N4IQLVPYPQ, seg_pos={0 0}, upload=334951]: Uploading piece... Dec 21 13:09:15.728917478 (381h40m15.600000513s) [noise=true , address=145.79.227.49:4337 , piece_id=DB3S7DUZSI5KIAKS7FQR6JHZEB3I7IDLBWEVJ2OHK7LOQ2OSSMXA, seg_pos={0 0}, upload=334951]: Uploading piece... Dec 21 13:09:15.729660793 (381h40m15.600743828s) [noise=true , address=145.79.227.50:4373 , piece_id=6BDNDSQ6WA4AVQFNL64FOKUCZ34R26ABBTP7QSZAJDSL6WV5AZBQ, seg_pos={0 0}, upload=334951]: Uploading piece... Dec 21 13:09:15.730010058 (381h40m15.601093093s) [noise=true , address=145.79.227.11:4731 , piece_id=AR427FU74URDWY6TBYNMKSTDE5VBSOB4L7YLCU3YSJLXOGJ4SV3A, seg_pos={0 0}, upload=334951]: Uploading piece... Dec 21 13:09:15.731038737 (381h40m15.602121772s) [noise=true , address=145.79.227.29:4942 , piece_id=Y52FXYDIGRHET2SKURCZPVZ5AKQH6P62YCZ5RCPWR6MC2ZMM7DGQ, seg_pos={0 0}, upload=334951]: Uploading piece... Dec 21 13:09:15.732070371 (381h40m15.603153416s) [noise=true , address=45.91.116.12:4227 , piece_id=U5BSL2YBQKMRTNGIRBCINKRD4OIA6VHKAPM6VZ3O4THDEJVABYKA, seg_pos={0 0}, upload=334951]: Uploading piece... Dec 21 13:09:15.733165196 (381h40m15.604248241s) [noise=true , address=145.79.227.14:4839 , piece_id=KM6HDO6CDKI4EGQ2YFCCWTXRMNDVGQD7DRT33JAJ63D4OHDHJSEQ, seg_pos={0 0}, upload=334951]: Uploading piece... Dec 21 13:09:17.607223723 (381h40m17.478306758s) [upload=334951]: Next returned nil segment Dec 21 13:09:17.607237008 (381h40m17.478320043s) [upload=334951]: Waiting for error group managing segments... Dec 21 13:09:17.927361266 (381h40m17.798444301s) [noise=true , address=145.79.227.37:4566 , piece_id=SNAWXEJ52STAJVCB6BXUXN4ETOIASJGIUS5IK4XEOZUYFDAXTJ3Q, seg_pos={0 0}, upload=334951]: Done uploading piece. err: <nil> Dec 21 13:09:17.964141351 (381h40m17.835224416s) [noise=true , address=145.79.227.47:4266 , piece_id=7SSF7C7LW4AYMNTLXRCI77OAWULVIWUUTPER6XLURIMML3CDZVGQ, seg_pos={0 0}, upload=334951]: Done uploading piece. err: <nil> Dec 21 13:09:17.982528162 (381h40m17.853611207s) [noise=true , address=145.79.227.11:4731 , piece_id=AR427FU74URDWY6TBYNMKSTDE5VBSOB4L7YLCU3YSJLXOGJ4SV3A, seg_pos={0 0}, upload=334951]: Done uploading piece. err: <nil> Dec 21 13:09:18.008304315 (381h40m17.87938736s) [noise=true , address=45.91.116.7:4012 , piece_id=OGMJVPHM74QXHBZLFRYWBNBTRBU5TYUINRP7MXBQMNT44AWUUEAA, seg_pos={0 0}, upload=334951]: Done uploading piece. err: <nil> Dec 21 13:09:18.022526526 (381h40m17.893609571s) [noise=true , address=45.91.116.12:4227 , piece_id=U5BSL2YBQKMRTNGIRBCINKRD4OIA6VHKAPM6VZ3O4THDEJVABYKA, seg_pos={0 0}, upload=334951]: Done uploading piece. err: <nil> Dec 21 13:09:18.352982186 (381h40m18.224065221s) [noise=true , address=45.91.116.10:4084 , piece_id=C5WX3TLQJFMTXZGTGXY3MRCOIQCNA3S6JKNDNNYRR5N4IQLVPYPQ, seg_pos={0 0}, upload=334951]: Done uploading piece. err: <nil> Dec 21 13:09:18.673062010 (381h40m18.544145055s) [noise=true , address=145.79.227.49:4337 , piece_id=DB3S7DUZSI5KIAKS7FQR6JHZEB3I7IDLBWEVJ2OHK7LOQ2OSSMXA, seg_pos={0 0}, upload=334951]: Done uploading piece. err: <nil> Dec 21 13:09:18.687869696 (381h40m18.558952741s) [noise=true , address=45.91.116.4:4939 , piece_id=7ROKKNPA22JAG3J4CMFI7WH2MB7DQFLPNL7GHZI4U27NO2RRP7DQ, seg_pos={0 0}, upload=334951]: Done uploading piece. err: <nil> Dec 21 13:09:18.877919140 (381h40m18.749002185s) [noise=true , address=145.79.227.20:4439 , piece_id=DEV552BLOMSZGVWSY3WDS4IMQNJ3PU2K3BSBSRVZCIWPD5NGR6FQ, seg_pos={0 0}, upload=334951]: Done uploading piece. err: <nil> Dec 21 13:09:18.913916405 (381h40m18.78499945s) [noise=true , address=145.79.227.46:4601 , piece_id=T5K4T3VUW3S43EKYTLBOU2ISNDXYMABN4O4IGYU2PDPOX47HKMEQ, seg_pos={0 0}, upload=334951]: Done uploading piece. err: <nil> Dec 21 13:09:18.932143993 (381h40m18.803227038s) [noise=true , address=145.79.227.32:4834 , piece_id=P6OD6IOQL5BP7MHSNYJWET7NABCA2HVDW7WLI52TYTTAAM4FL3XA, seg_pos={0 0}, upload=334951]: Done uploading piece. err: <nil> Dec 21 13:09:19.011779392 (381h40m18.882862427s) [noise=true , address=145.79.227.14:4839 , piece_id=KM6HDO6CDKI4EGQ2YFCCWTXRMNDVGQD7DRT33JAJ63D4OHDHJSEQ, seg_pos={0 0}, upload=334951]: Done uploading piece. err: <nil> Dec 21 13:09:19.334088539 (381h40m19.205171574s) [noise=true , address=145.79.227.29:4942 , piece_id=Y52FXYDIGRHET2SKURCZPVZ5AKQH6P62YCZ5RCPWR6MC2ZMM7DGQ, seg_pos={0 0}, upload=334951]: Done uploading piece. err: <nil> Dec 21 13:09:19.334107215 (381h40m19.20519025s) [seg_pos={0 0}, upload=334951]: Segment reached optimal threshold of 13 pieces. Dec 21 13:09:19.334276458 (381h40m19.205359493s) [noise=true , address=145.79.227.51:4411 , piece_id=EW5M23F4QLXZCVGVHHEVWH74BVXZNPS7JEBEAO6PJK7PEDTN4LXA, seg_pos={0 0}, upload=334951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR): protocol: expected piece hash; context canceled; EOF Dec 21 13:09:19.334283371 (381h40m19.205366416s) [noise=true , address=145.79.227.51:4411 , piece_id=EW5M23F4QLXZCVGVHHEVWH74BVXZNPS7JEBEAO6PJK7PEDTN4LXA, seg_pos={0 0}, upload=334951]: Long tail cancellation due to optimal threshold Dec 21 13:09:19.334576568 (381h40m19.205659604s) [noise=true , address=145.79.227.50:4373 , piece_id=6BDNDSQ6WA4AVQFNL64FOKUCZ34R26ABBTP7QSZAJDSL6WV5AZBQ, seg_pos={0 0}, upload=334951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: expected piece hash; context canceled; EOF Dec 21 13:09:19.334582891 (381h40m19.205665926s) [noise=true , address=145.79.227.50:4373 , piece_id=6BDNDSQ6WA4AVQFNL64FOKUCZ34R26ABBTP7QSZAJDSL6WV5AZBQ, seg_pos={0 0}, upload=334951]: Long tail cancellation due to optimal threshold Dec 21 13:09:19.334684755 (381h40m19.20576779s) [noise=true , address=145.79.227.16:4911 , piece_id=T3IVA3SR72JPXQERVZQPFHIAIHJDG6PIQHX2QPBY4LE7T2AI6EYA, seg_pos={0 0}, upload=334951]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: expected piece hash; context canceled; EOF Dec 21 13:09:19.334691438 (381h40m19.205774473s) [noise=true , address=145.79.227.16:4911 , piece_id=T3IVA3SR72JPXQERVZQPFHIAIHJDG6PIQHX2QPBY4LE7T2AI6EYA, seg_pos={0 0}, upload=334951]: Long tail cancellation due to optimal threshold Dec 21 13:09:19.334708770 (381h40m19.205791805s) [seg_pos={0 0}, upload=334951]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 13:09:19.334774997 (381h40m19.205858032s) [upload=334951]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 13:09:19.334779265 (381h40m19.2058623s) [upload=334951]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 13:09:19.631682616 (381h40m19.502765651s) [upload=334951]: Done uploading segments... Dec 21 13:09:19.631727181 (381h40m19.502810226s) [upload=334951]: Upload finished. err: <nil> Duration: 30.004801396s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.X.X.X.....X..X.....X..X....X........X.....................X] duration: 30.00288677s trace: https://jaeger.dev.storj.io/trace/51c81ddae717fbcf FAIL (2025-12-21 13:09:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, Piece ID: BFSSRDZ5C3QTAFJIXDDTGN6G42LCSNQT77YHNX465EAZVXQH3IHQ) context deadline exceeded; piecestore: (Node ID: 12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, Piece ID: 3RZZNPO2OKZ2LMIBQS5GSVH7EDXV5CXJFW3BS4M4EVWENSHPJB2Q) context deadline exceeded; piecestore: (Node ID: 1gYjGpNnkeFRobb7yhMnm1o9N3H13Uo1fqQrBpRad4uU4rKKm1, Piece ID: FSNWDTV5P6GJXCWQW47WTEBPZ5DMX7VYNBZDQ5IPXPDICDRCGCAA) context deadline exceeded ======LOGS====== 3:08:45.305956687 (381h39m45.177039722s) [upload=334944]: Starting upload Dec 21 13:08:45.306124065 (381h39m45.1772071s) [upload=334944]: Done starting upload Dec 21 13:08:45.306224717 (381h39m45.177307752s) [upload=334944]: Uploading segments... Dec 21 13:08:45.306627995 (381h39m45.17771103s) [upload=334944]: Got next segment. Inline: false Dec 21 13:08:45.306678080 (381h39m45.177761125s) [upload=334944]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 13:08:45.306684112 (381h39m45.177767147s) [upload=334944]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 13:08:45.819538698 (381h39m45.690621753s) [seg_pos={0 0}, upload=334944]: Begin upload segment... Dec 21 13:08:45.819596508 (381h39m45.690679553s) [seg_pos={0 0}, upload=334944]: Done begin upload segment. Dec 21 13:08:45.819628409 (381h39m45.690711444s) [noise=true , address=145.79.227.10:4401 , piece_id=FSNWDTV5P6GJXCWQW47WTEBPZ5DMX7VYNBZDQ5IPXPDICDRCGCAA, seg_pos={0 0}, upload=334944]: Uploading piece... Dec 21 13:08:45.819745041 (381h39m45.690828086s) [noise=true , address=145.79.227.41:4085 , piece_id=FCDNLHD6UDIWRKOLSP3ALTX3U7RZB3B7ZVHUYDGRAN2VVKCI24DQ, seg_pos={0 0}, upload=334944]: Uploading piece... Dec 21 13:08:45.820297253 (381h39m45.691380298s) [noise=true , address=145.79.227.47:4267 , piece_id=BFSSRDZ5C3QTAFJIXDDTGN6G42LCSNQT77YHNX465EAZVXQH3IHQ, seg_pos={0 0}, upload=334944]: Uploading piece... Dec 21 13:08:45.820866787 (381h39m45.691949822s) [noise=true , address=145.79.227.42:4123 , piece_id=PZGBYNCWEZLBFDZBMYAZNXTW3HMRUG5LLJIRGXJKEVAOIAGFJQKQ, seg_pos={0 0}, upload=334944]: Uploading piece... Dec 21 13:08:45.821069243 (381h39m45.692152278s) [noise=true , address=145.79.227.25:4617 , piece_id=6M6VH7TLBR2BC2FPPVLTW45BF6BJ6LPKDHPJZ4NY65FWR7XANWKQ, seg_pos={0 0}, upload=334944]: Uploading piece... Dec 21 13:08:45.821208979 (381h39m45.692292014s) [noise=true , address=145.79.227.29:4941 , piece_id=WLNA2V5N5LXIZWVTDJUKEH2E6FZ7X5O2HZRC5OH5T7ADB7EXTLRA, seg_pos={0 0}, upload=334944]: Uploading piece... Dec 21 13:08:45.821509220 (381h39m45.692592255s) [noise=true , address=145.79.227.37:4565 , piece_id=V27XFW3C274NEDSGXQAIVCJO4YR7JWK6IIZJITZPPIMHJQTCVDKQ, seg_pos={0 0}, upload=334944]: Uploading piece... Dec 21 13:08:45.821536492 (381h39m45.692619537s) [noise=true , address=145.79.227.48:4303 , piece_id=CXCU2YUVMLCNQBQ35NJM442OHAQ66NUSGO4YQX2AKWPV5ONXXFHQ, seg_pos={0 0}, upload=334944]: Uploading piece... Dec 21 13:08:45.821768374 (381h39m45.692851419s) [noise=true , address=145.79.227.8:4330 , piece_id=JFYJ75HZ42WZWYF44QJLTDN6LQHULBGCOZFWHJY7AUNMIILI5MMA, seg_pos={0 0}, upload=334944]: Uploading piece... Dec 21 13:08:45.822129642 (381h39m45.693212687s) [noise=true , address=145.79.227.34:4905 , piece_id=52JKTZH6MG3AX2ABJCTRCWX6KFV3F6FF5ZZMC5Q36SJJFEXG7DEA, seg_pos={0 0}, upload=334944]: Uploading piece... Dec 21 13:08:45.822190578 (381h39m45.693273613s) [noise=true , address=45.91.116.8:4045 , piece_id=DXQUDOSDCOS66VLBR6EPQDUX55DFKZ3DMMKOZTUXTX6TRYLBTOJQ, seg_pos={0 0}, upload=334944]: Uploading piece... Dec 21 13:08:45.822487883 (381h39m45.693570928s) [noise=true , address=145.79.227.35:4481 , piece_id=3RZZNPO2OKZ2LMIBQS5GSVH7EDXV5CXJFW3BS4M4EVWENSHPJB2Q, seg_pos={0 0}, upload=334944]: Uploading piece... Dec 21 13:08:45.822639012 (381h39m45.693722057s) [noise=true , address=145.79.227.15:4875 , piece_id=NGCTY4TINRC5BF6KYXVNBBD2DPFNPKDNVUQY3WKSPGQSJCFO4RYQ, seg_pos={0 0}, upload=334944]: Uploading piece... Dec 21 13:08:45.823134936 (381h39m45.694217971s) [noise=true , address=45.91.116.6:4976 , piece_id=4QQLMYSCHZLQAR43IMHJYO7E2ZXXDJY3CJQCHECA7AFEX4TZCFVA, seg_pos={0 0}, upload=334944]: Uploading piece... Dec 21 13:08:45.823662470 (381h39m45.694745515s) [noise=true , address=77.90.31.13:4629 , piece_id=ELTJBGEHQ3CQIMUBYNWSQRDU6OE7J3LN257TWKNROZ46MR2BUJJQ, seg_pos={0 0}, upload=334944]: Uploading piece... Dec 21 13:08:45.824174936 (381h39m45.695257981s) [noise=true , address=145.79.227.51:4409 , piece_id=HKR4GJMQG5JMTU4SGEFD3B5NUC6XJRFVUZGKXIJP2445XDWTTY4A, seg_pos={0 0}, upload=334944]: Uploading piece... Dec 21 13:08:47.677409310 (381h39m47.548492345s) [upload=334944]: Next returned nil segment Dec 21 13:08:47.677422124 (381h39m47.548505169s) [upload=334944]: Waiting for error group managing segments... Dec 21 13:08:47.974371514 (381h39m47.845454559s) [noise=true , address=145.79.227.42:4123 , piece_id=PZGBYNCWEZLBFDZBMYAZNXTW3HMRUG5LLJIRGXJKEVAOIAGFJQKQ, seg_pos={0 0}, upload=334944]: Done uploading piece. err: <nil> Dec 21 13:08:47.987525901 (381h39m47.858608936s) [noise=true , address=145.79.227.8:4330 , piece_id=JFYJ75HZ42WZWYF44QJLTDN6LQHULBGCOZFWHJY7AUNMIILI5MMA, seg_pos={0 0}, upload=334944]: Done uploading piece. err: <nil> Dec 21 13:08:48.009099641 (381h39m47.880182677s) [noise=true , address=145.79.227.10:4401 , piece_id=FSNWDTV5P6GJXCWQW47WTEBPZ5DMX7VYNBZDQ5IPXPDICDRCGCAA, seg_pos={0 0}, upload=334944]: Done uploading piece. err: <nil> Dec 21 13:08:48.018226961 (381h39m47.889309996s) [noise=true , address=145.79.227.47:4267 , piece_id=BFSSRDZ5C3QTAFJIXDDTGN6G42LCSNQT77YHNX465EAZVXQH3IHQ, seg_pos={0 0}, upload=334944]: Done uploading piece. err: <nil> Dec 21 13:08:48.054907557 (381h39m47.925990602s) [noise=true , address=145.79.227.51:4409 , piece_id=HKR4GJMQG5JMTU4SGEFD3B5NUC6XJRFVUZGKXIJP2445XDWTTY4A, seg_pos={0 0}, upload=334944]: Done uploading piece. err: <nil> Dec 21 13:08:48.073685332 (381h39m47.944768367s) [noise=true , address=145.79.227.35:4481 , piece_id=3RZZNPO2OKZ2LMIBQS5GSVH7EDXV5CXJFW3BS4M4EVWENSHPJB2Q, seg_pos={0 0}, upload=334944]: Done uploading piece. err: <nil> Dec 21 13:08:48.096098681 (381h39m47.967181716s) [noise=true , address=45.91.116.6:4976 , piece_id=4QQLMYSCHZLQAR43IMHJYO7E2ZXXDJY3CJQCHECA7AFEX4TZCFVA, seg_pos={0 0}, upload=334944]: Done uploading piece. err: <nil> Dec 21 13:08:48.110747065 (381h39m47.9818301s) [noise=true , address=77.90.31.13:4629 , piece_id=ELTJBGEHQ3CQIMUBYNWSQRDU6OE7J3LN257TWKNROZ46MR2BUJJQ, seg_pos={0 0}, upload=334944]: Done uploading piece. err: <nil> Dec 21 13:08:48.119674313 (381h39m47.990757348s) [noise=true , address=45.91.116.8:4045 , piece_id=DXQUDOSDCOS66VLBR6EPQDUX55DFKZ3DMMKOZTUXTX6TRYLBTOJQ, seg_pos={0 0}, upload=334944]: Done uploading piece. err: <nil> Dec 21 13:08:48.334843541 (381h39m48.205926586s) [noise=true , address=145.79.227.15:4875 , piece_id=NGCTY4TINRC5BF6KYXVNBBD2DPFNPKDNVUQY3WKSPGQSJCFO4RYQ, seg_pos={0 0}, upload=334944]: Done uploading piece. err: <nil> Dec 21 13:08:48.670978836 (381h39m48.542061871s) [noise=true , address=145.79.227.29:4941 , piece_id=WLNA2V5N5LXIZWVTDJUKEH2E6FZ7X5O2HZRC5OH5T7ADB7EXTLRA, seg_pos={0 0}, upload=334944]: Done uploading piece. err: <nil> Dec 21 13:08:49.005413060 (381h39m48.876496115s) [noise=true , address=145.79.227.41:4085 , piece_id=FCDNLHD6UDIWRKOLSP3ALTX3U7RZB3B7ZVHUYDGRAN2VVKCI24DQ, seg_pos={0 0}, upload=334944]: Done uploading piece. err: <nil> Dec 21 13:08:49.121357388 (381h39m48.992440423s) [noise=true , address=145.79.227.48:4303 , piece_id=CXCU2YUVMLCNQBQ35NJM442OHAQ66NUSGO4YQX2AKWPV5ONXXFHQ, seg_pos={0 0}, upload=334944]: Done uploading piece. err: <nil> Dec 21 13:08:49.121376954 (381h39m48.992459989s) [seg_pos={0 0}, upload=334944]: Segment reached optimal threshold of 13 pieces. Dec 21 13:08:49.121557489 (381h39m48.992640524s) [noise=true , address=145.79.227.37:4565 , piece_id=V27XFW3C274NEDSGXQAIVCJO4YR7JWK6IIZJITZPPIMHJQTCVDKQ, seg_pos={0 0}, upload=334944]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT): protocol: expected piece hash; context canceled; EOF Dec 21 13:08:49.121563761 (381h39m48.992646806s) [noise=true , address=145.79.227.37:4565 , piece_id=V27XFW3C274NEDSGXQAIVCJO4YR7JWK6IIZJITZPPIMHJQTCVDKQ, seg_pos={0 0}, upload=334944]: Long tail cancellation due to optimal threshold Dec 21 13:08:49.121686013 (381h39m48.992769048s) [noise=true , address=145.79.227.34:4905 , piece_id=52JKTZH6MG3AX2ABJCTRCWX6KFV3F6FF5ZZMC5Q36SJJFEXG7DEA, seg_pos={0 0}, upload=334944]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 21 13:08:49.121693106 (381h39m48.992776151s) [noise=true , address=145.79.227.34:4905 , piece_id=52JKTZH6MG3AX2ABJCTRCWX6KFV3F6FF5ZZMC5Q36SJJFEXG7DEA, seg_pos={0 0}, upload=334944]: Long tail cancellation due to optimal threshold Dec 21 13:08:49.121941550 (381h39m48.993024595s) [noise=true , address=145.79.227.25:4617 , piece_id=6M6VH7TLBR2BC2FPPVLTW45BF6BJ6LPKDHPJZ4NY65FWR7XANWKQ, seg_pos={0 0}, upload=334944]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: write tcp 188.166.225.170:53394->145.79.227.25:4617: use of closed network connection; write tcp 188.166.225.170:53394->145.79.227.25:4617: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53394->145.79.227.25:4617: use of closed network connection Dec 21 13:08:49.121949475 (381h39m48.99303251s) [noise=true , address=145.79.227.25:4617 , piece_id=6M6VH7TLBR2BC2FPPVLTW45BF6BJ6LPKDHPJZ4NY65FWR7XANWKQ, seg_pos={0 0}, upload=334944]: Long tail cancellation due to optimal threshold Dec 21 13:08:49.122053884 (381h39m48.993136919s) [seg_pos={0 0}, upload=334944]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 13:08:49.122134607 (381h39m48.993217642s) [upload=334944]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 13:08:49.122139276 (381h39m48.993222321s) [upload=334944]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 13:08:50.185559607 (381h39m50.056642672s) [upload=334944]: Done uploading segments... Dec 21 13:08:50.185618930 (381h39m50.056701965s) [upload=334944]: Upload finished. err: <nil> Duration: 30.002911196s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..X.X.X.....X..X.....X..X....X........X.....................] duration: 5.91714005s trace: https://jaeger.dev.storj.io/trace/5254881e4ca2c0fe FAIL (2025-12-21 13:08:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, Piece ID: YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A) context deadline exceeded; piecestore: (Node ID: 1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, Piece ID: I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ) context deadline exceeded; piecestore: (Node ID: 12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, Piece ID: 7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ) context deadline exceeded; piecestore: (Node ID: 17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, Piece ID: 36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q) context deadline exceeded ======LOGS====== ents... Dec 21 12:53:32.080943203 (381h24m31.952026248s) [upload=334718]: Got next segment. Inline: false Dec 21 12:53:32.080975975 (381h24m31.95205901s) [upload=334718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:53:32.080980384 (381h24m31.952063419s) [upload=334718]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:53:32.328092505 (381h24m32.19917554s) [seg_pos={0 0}, upload=334718]: Begin upload segment... Dec 21 12:53:32.328182436 (381h24m32.199265471s) [seg_pos={0 0}, upload=334718]: Done begin upload segment. Dec 21 12:53:32.328220579 (381h24m32.199303624s) [noise=true , address=145.79.227.48:4302 , piece_id=GIFTHYKLR2K3DJLHB4VZ6T7NMYSA5TRAKPTEEJ53UVMQTYROMECA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.331668685 (381h24m32.20275173s) [noise=true , address=145.79.227.52:4445 , piece_id=SJDQEFSKBN6ERSH2WJUKMY4TEKA2XJQHDU7YBGTZWB7OQ52WE3KA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.332500137 (381h24m32.203583172s) [noise=true , address=145.79.227.47:4267 , piece_id=YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.333277057 (381h24m32.204360092s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.334224581 (381h24m32.205307617s) [noise=true , address=145.79.227.42:4121 , piece_id=PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.334844001 (381h24m32.205927046s) [noise=true , address=45.91.116.6:4974 , piece_id=ZX3KN4IETHLW3ABY6J63I35OMTQC2ZMFKO3GBPU4PY4HDS6OA22A, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.335828375 (381h24m32.20691142s) [noise=true , address=145.79.227.29:4943 , piece_id=DZZBLVTCTKGAHIFCHYNS5JLXQD2IBSP6XELBRGWS4QUBPYFMH7SA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.336610345 (381h24m32.20769338s) [noise=true , address=145.79.227.49:4338 , piece_id=I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.337469029 (381h24m32.208552074s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.338354977 (381h24m32.209438022s) [noise=true , address=145.79.227.34:4907 , piece_id=7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.339032116 (381h24m32.210115161s) [noise=true , address=145.79.227.15:4875 , piece_id=J53SUZ5OSH7B6THTLOG6762X2VDONOODZQSPT4I3BKGOCZZV3HEA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.340261366 (381h24m32.211344401s) [noise=true , address=145.79.227.14:4839 , piece_id=RMPSFFROF52HBMWI7HYYOZRYPN3YM6CCUSZJQABGOCKQOEOMGPZA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.340406363 (381h24m32.211489398s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.341633339 (381h24m32.212716374s) [noise=true , address=145.79.227.33:4870 , piece_id=GO6YIYMOGWGKS3KHIKAUBTOYTJLENKH6M7HJWLHC3OTNMUA4WJXA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.342100158 (381h24m32.213183193s) [noise=true , address=145.79.227.41:4085 , piece_id=36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.343690196 (381h24m32.214773241s) [noise=true , address=145.79.227.32:4834 , piece_id=T6AGJLKTKW4MJTMXEBAW4YBF5BSWQRKHWYKLEU4Y5U37NAMMAJTQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:34.215235315 (381h24m34.08631835s) [upload=334718]: Next returned nil segment Dec 21 12:53:34.215255653 (381h24m34.086338689s) [upload=334718]: Waiting for error group managing segments... Dec 21 12:53:34.523622594 (381h24m34.394705629s) [noise=true , address=145.79.227.49:4338 , piece_id=I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.525892566 (381h24m34.396975601s) [noise=true , address=145.79.227.14:4839 , piece_id=RMPSFFROF52HBMWI7HYYOZRYPN3YM6CCUSZJQABGOCKQOEOMGPZA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.535347309 (381h24m34.406430344s) [noise=true , address=145.79.227.41:4085 , piece_id=36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.592006026 (381h24m34.463089061s) [noise=true , address=145.79.227.34:4907 , piece_id=7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.609875664 (381h24m34.480958709s) [noise=true , address=145.79.227.32:4834 , piece_id=T6AGJLKTKW4MJTMXEBAW4YBF5BSWQRKHWYKLEU4Y5U37NAMMAJTQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.622134556 (381h24m34.493217591s) [noise=true , address=145.79.227.15:4875 , piece_id=J53SUZ5OSH7B6THTLOG6762X2VDONOODZQSPT4I3BKGOCZZV3HEA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.755361442 (381h24m34.626444487s) [noise=true , address=145.79.227.29:4943 , piece_id=DZZBLVTCTKGAHIFCHYNS5JLXQD2IBSP6XELBRGWS4QUBPYFMH7SA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.876619305 (381h24m34.74770235s) [noise=true , address=145.79.227.33:4870 , piece_id=GO6YIYMOGWGKS3KHIKAUBTOYTJLENKH6M7HJWLHC3OTNMUA4WJXA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.899187028 (381h24m34.770270073s) [noise=true , address=145.79.227.52:4445 , piece_id=SJDQEFSKBN6ERSH2WJUKMY4TEKA2XJQHDU7YBGTZWB7OQ52WE3KA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.205676652 (381h24m35.076759698s) [noise=true , address=145.79.227.42:4121 , piece_id=PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.259291003 (381h24m35.130374048s) [noise=true , address=145.79.227.48:4302 , piece_id=GIFTHYKLR2K3DJLHB4VZ6T7NMYSA5TRAKPTEEJ53UVMQTYROMECA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.326071153 (381h24m35.197154188s) [noise=true , address=45.91.116.6:4974 , piece_id=ZX3KN4IETHLW3ABY6J63I35OMTQC2ZMFKO3GBPU4PY4HDS6OA22A, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.502232413 (381h24m35.373315448s) [noise=true , address=145.79.227.47:4267 , piece_id=YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.502253854 (381h24m35.373336919s) [seg_pos={0 0}, upload=334718]: Segment reached optimal threshold of 13 pieces. Dec 21 12:53:35.502568303 (381h24m35.373651338s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 21 12:53:35.502581327 (381h24m35.373664372s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502608810 (381h24m35.373691845s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection Dec 21 12:53:35.502619169 (381h24m35.373702214s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502846553 (381h24m35.373929588s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection Dec 21 12:53:35.502853666 (381h24m35.373936701s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502873894 (381h24m35.373956929s) [seg_pos={0 0}, upload=334718]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:53:35.502965338 (381h24m35.374048383s) [upload=334718]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:53:35.502970939 (381h24m35.374053984s) [upload=334718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:53:35.794077451 (381h24m35.665160486s) [upload=334718]: Done uploading segments... Dec 21 12:53:35.794103912 (381h24m35.665186957s) [upload=334718]: Upload finished. err: <nil> Duration: 5.917168263s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...X.X.X.....X..X.....X..X....X........X....................] duration: 5.750426903s trace: https://jaeger.dev.storj.io/trace/5254881e4ca2c0fe FAIL (2025-12-21 13:07:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, Piece ID: YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A) context deadline exceeded; piecestore: (Node ID: 1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, Piece ID: I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ) context deadline exceeded; piecestore: (Node ID: 12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, Piece ID: 7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ) context deadline exceeded; piecestore: (Node ID: 17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, Piece ID: 36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q) context deadline exceeded ======LOGS====== ents... Dec 21 12:53:32.080943203 (381h24m31.952026248s) [upload=334718]: Got next segment. Inline: false Dec 21 12:53:32.080975975 (381h24m31.95205901s) [upload=334718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:53:32.080980384 (381h24m31.952063419s) [upload=334718]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:53:32.328092505 (381h24m32.19917554s) [seg_pos={0 0}, upload=334718]: Begin upload segment... Dec 21 12:53:32.328182436 (381h24m32.199265471s) [seg_pos={0 0}, upload=334718]: Done begin upload segment. Dec 21 12:53:32.328220579 (381h24m32.199303624s) [noise=true , address=145.79.227.48:4302 , piece_id=GIFTHYKLR2K3DJLHB4VZ6T7NMYSA5TRAKPTEEJ53UVMQTYROMECA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.331668685 (381h24m32.20275173s) [noise=true , address=145.79.227.52:4445 , piece_id=SJDQEFSKBN6ERSH2WJUKMY4TEKA2XJQHDU7YBGTZWB7OQ52WE3KA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.332500137 (381h24m32.203583172s) [noise=true , address=145.79.227.47:4267 , piece_id=YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.333277057 (381h24m32.204360092s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.334224581 (381h24m32.205307617s) [noise=true , address=145.79.227.42:4121 , piece_id=PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.334844001 (381h24m32.205927046s) [noise=true , address=45.91.116.6:4974 , piece_id=ZX3KN4IETHLW3ABY6J63I35OMTQC2ZMFKO3GBPU4PY4HDS6OA22A, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.335828375 (381h24m32.20691142s) [noise=true , address=145.79.227.29:4943 , piece_id=DZZBLVTCTKGAHIFCHYNS5JLXQD2IBSP6XELBRGWS4QUBPYFMH7SA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.336610345 (381h24m32.20769338s) [noise=true , address=145.79.227.49:4338 , piece_id=I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.337469029 (381h24m32.208552074s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.338354977 (381h24m32.209438022s) [noise=true , address=145.79.227.34:4907 , piece_id=7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.339032116 (381h24m32.210115161s) [noise=true , address=145.79.227.15:4875 , piece_id=J53SUZ5OSH7B6THTLOG6762X2VDONOODZQSPT4I3BKGOCZZV3HEA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.340261366 (381h24m32.211344401s) [noise=true , address=145.79.227.14:4839 , piece_id=RMPSFFROF52HBMWI7HYYOZRYPN3YM6CCUSZJQABGOCKQOEOMGPZA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.340406363 (381h24m32.211489398s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.341633339 (381h24m32.212716374s) [noise=true , address=145.79.227.33:4870 , piece_id=GO6YIYMOGWGKS3KHIKAUBTOYTJLENKH6M7HJWLHC3OTNMUA4WJXA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.342100158 (381h24m32.213183193s) [noise=true , address=145.79.227.41:4085 , piece_id=36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.343690196 (381h24m32.214773241s) [noise=true , address=145.79.227.32:4834 , piece_id=T6AGJLKTKW4MJTMXEBAW4YBF5BSWQRKHWYKLEU4Y5U37NAMMAJTQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:34.215235315 (381h24m34.08631835s) [upload=334718]: Next returned nil segment Dec 21 12:53:34.215255653 (381h24m34.086338689s) [upload=334718]: Waiting for error group managing segments... Dec 21 12:53:34.523622594 (381h24m34.394705629s) [noise=true , address=145.79.227.49:4338 , piece_id=I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.525892566 (381h24m34.396975601s) [noise=true , address=145.79.227.14:4839 , piece_id=RMPSFFROF52HBMWI7HYYOZRYPN3YM6CCUSZJQABGOCKQOEOMGPZA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.535347309 (381h24m34.406430344s) [noise=true , address=145.79.227.41:4085 , piece_id=36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.592006026 (381h24m34.463089061s) [noise=true , address=145.79.227.34:4907 , piece_id=7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.609875664 (381h24m34.480958709s) [noise=true , address=145.79.227.32:4834 , piece_id=T6AGJLKTKW4MJTMXEBAW4YBF5BSWQRKHWYKLEU4Y5U37NAMMAJTQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.622134556 (381h24m34.493217591s) [noise=true , address=145.79.227.15:4875 , piece_id=J53SUZ5OSH7B6THTLOG6762X2VDONOODZQSPT4I3BKGOCZZV3HEA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.755361442 (381h24m34.626444487s) [noise=true , address=145.79.227.29:4943 , piece_id=DZZBLVTCTKGAHIFCHYNS5JLXQD2IBSP6XELBRGWS4QUBPYFMH7SA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.876619305 (381h24m34.74770235s) [noise=true , address=145.79.227.33:4870 , piece_id=GO6YIYMOGWGKS3KHIKAUBTOYTJLENKH6M7HJWLHC3OTNMUA4WJXA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.899187028 (381h24m34.770270073s) [noise=true , address=145.79.227.52:4445 , piece_id=SJDQEFSKBN6ERSH2WJUKMY4TEKA2XJQHDU7YBGTZWB7OQ52WE3KA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.205676652 (381h24m35.076759698s) [noise=true , address=145.79.227.42:4121 , piece_id=PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.259291003 (381h24m35.130374048s) [noise=true , address=145.79.227.48:4302 , piece_id=GIFTHYKLR2K3DJLHB4VZ6T7NMYSA5TRAKPTEEJ53UVMQTYROMECA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.326071153 (381h24m35.197154188s) [noise=true , address=45.91.116.6:4974 , piece_id=ZX3KN4IETHLW3ABY6J63I35OMTQC2ZMFKO3GBPU4PY4HDS6OA22A, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.502232413 (381h24m35.373315448s) [noise=true , address=145.79.227.47:4267 , piece_id=YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.502253854 (381h24m35.373336919s) [seg_pos={0 0}, upload=334718]: Segment reached optimal threshold of 13 pieces. Dec 21 12:53:35.502568303 (381h24m35.373651338s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 21 12:53:35.502581327 (381h24m35.373664372s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502608810 (381h24m35.373691845s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection Dec 21 12:53:35.502619169 (381h24m35.373702214s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502846553 (381h24m35.373929588s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection Dec 21 12:53:35.502853666 (381h24m35.373936701s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502873894 (381h24m35.373956929s) [seg_pos={0 0}, upload=334718]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:53:35.502965338 (381h24m35.374048383s) [upload=334718]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:53:35.502970939 (381h24m35.374053984s) [upload=334718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:53:35.794077451 (381h24m35.665160486s) [upload=334718]: Done uploading segments... Dec 21 12:53:35.794103912 (381h24m35.665186957s) [upload=334718]: Upload finished. err: <nil> Duration: 5.75046777s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [....X.X.X.....X..X.....X..X....X........X...................] duration: 5.894190179s trace: https://jaeger.dev.storj.io/trace/5254881e4ca2c0fe FAIL (2025-12-21 13:07:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, Piece ID: YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A) context deadline exceeded; piecestore: (Node ID: 1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, Piece ID: I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ) context deadline exceeded; piecestore: (Node ID: 12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, Piece ID: 7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ) context deadline exceeded; piecestore: (Node ID: 17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, Piece ID: 36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q) context deadline exceeded ======LOGS====== ents... Dec 21 12:53:32.080943203 (381h24m31.952026248s) [upload=334718]: Got next segment. Inline: false Dec 21 12:53:32.080975975 (381h24m31.95205901s) [upload=334718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:53:32.080980384 (381h24m31.952063419s) [upload=334718]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:53:32.328092505 (381h24m32.19917554s) [seg_pos={0 0}, upload=334718]: Begin upload segment... Dec 21 12:53:32.328182436 (381h24m32.199265471s) [seg_pos={0 0}, upload=334718]: Done begin upload segment. Dec 21 12:53:32.328220579 (381h24m32.199303624s) [noise=true , address=145.79.227.48:4302 , piece_id=GIFTHYKLR2K3DJLHB4VZ6T7NMYSA5TRAKPTEEJ53UVMQTYROMECA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.331668685 (381h24m32.20275173s) [noise=true , address=145.79.227.52:4445 , piece_id=SJDQEFSKBN6ERSH2WJUKMY4TEKA2XJQHDU7YBGTZWB7OQ52WE3KA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.332500137 (381h24m32.203583172s) [noise=true , address=145.79.227.47:4267 , piece_id=YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.333277057 (381h24m32.204360092s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.334224581 (381h24m32.205307617s) [noise=true , address=145.79.227.42:4121 , piece_id=PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.334844001 (381h24m32.205927046s) [noise=true , address=45.91.116.6:4974 , piece_id=ZX3KN4IETHLW3ABY6J63I35OMTQC2ZMFKO3GBPU4PY4HDS6OA22A, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.335828375 (381h24m32.20691142s) [noise=true , address=145.79.227.29:4943 , piece_id=DZZBLVTCTKGAHIFCHYNS5JLXQD2IBSP6XELBRGWS4QUBPYFMH7SA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.336610345 (381h24m32.20769338s) [noise=true , address=145.79.227.49:4338 , piece_id=I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.337469029 (381h24m32.208552074s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.338354977 (381h24m32.209438022s) [noise=true , address=145.79.227.34:4907 , piece_id=7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.339032116 (381h24m32.210115161s) [noise=true , address=145.79.227.15:4875 , piece_id=J53SUZ5OSH7B6THTLOG6762X2VDONOODZQSPT4I3BKGOCZZV3HEA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.340261366 (381h24m32.211344401s) [noise=true , address=145.79.227.14:4839 , piece_id=RMPSFFROF52HBMWI7HYYOZRYPN3YM6CCUSZJQABGOCKQOEOMGPZA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.340406363 (381h24m32.211489398s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.341633339 (381h24m32.212716374s) [noise=true , address=145.79.227.33:4870 , piece_id=GO6YIYMOGWGKS3KHIKAUBTOYTJLENKH6M7HJWLHC3OTNMUA4WJXA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.342100158 (381h24m32.213183193s) [noise=true , address=145.79.227.41:4085 , piece_id=36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.343690196 (381h24m32.214773241s) [noise=true , address=145.79.227.32:4834 , piece_id=T6AGJLKTKW4MJTMXEBAW4YBF5BSWQRKHWYKLEU4Y5U37NAMMAJTQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:34.215235315 (381h24m34.08631835s) [upload=334718]: Next returned nil segment Dec 21 12:53:34.215255653 (381h24m34.086338689s) [upload=334718]: Waiting for error group managing segments... Dec 21 12:53:34.523622594 (381h24m34.394705629s) [noise=true , address=145.79.227.49:4338 , piece_id=I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.525892566 (381h24m34.396975601s) [noise=true , address=145.79.227.14:4839 , piece_id=RMPSFFROF52HBMWI7HYYOZRYPN3YM6CCUSZJQABGOCKQOEOMGPZA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.535347309 (381h24m34.406430344s) [noise=true , address=145.79.227.41:4085 , piece_id=36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.592006026 (381h24m34.463089061s) [noise=true , address=145.79.227.34:4907 , piece_id=7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.609875664 (381h24m34.480958709s) [noise=true , address=145.79.227.32:4834 , piece_id=T6AGJLKTKW4MJTMXEBAW4YBF5BSWQRKHWYKLEU4Y5U37NAMMAJTQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.622134556 (381h24m34.493217591s) [noise=true , address=145.79.227.15:4875 , piece_id=J53SUZ5OSH7B6THTLOG6762X2VDONOODZQSPT4I3BKGOCZZV3HEA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.755361442 (381h24m34.626444487s) [noise=true , address=145.79.227.29:4943 , piece_id=DZZBLVTCTKGAHIFCHYNS5JLXQD2IBSP6XELBRGWS4QUBPYFMH7SA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.876619305 (381h24m34.74770235s) [noise=true , address=145.79.227.33:4870 , piece_id=GO6YIYMOGWGKS3KHIKAUBTOYTJLENKH6M7HJWLHC3OTNMUA4WJXA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.899187028 (381h24m34.770270073s) [noise=true , address=145.79.227.52:4445 , piece_id=SJDQEFSKBN6ERSH2WJUKMY4TEKA2XJQHDU7YBGTZWB7OQ52WE3KA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.205676652 (381h24m35.076759698s) [noise=true , address=145.79.227.42:4121 , piece_id=PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.259291003 (381h24m35.130374048s) [noise=true , address=145.79.227.48:4302 , piece_id=GIFTHYKLR2K3DJLHB4VZ6T7NMYSA5TRAKPTEEJ53UVMQTYROMECA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.326071153 (381h24m35.197154188s) [noise=true , address=45.91.116.6:4974 , piece_id=ZX3KN4IETHLW3ABY6J63I35OMTQC2ZMFKO3GBPU4PY4HDS6OA22A, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.502232413 (381h24m35.373315448s) [noise=true , address=145.79.227.47:4267 , piece_id=YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.502253854 (381h24m35.373336919s) [seg_pos={0 0}, upload=334718]: Segment reached optimal threshold of 13 pieces. Dec 21 12:53:35.502568303 (381h24m35.373651338s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 21 12:53:35.502581327 (381h24m35.373664372s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502608810 (381h24m35.373691845s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection Dec 21 12:53:35.502619169 (381h24m35.373702214s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502846553 (381h24m35.373929588s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection Dec 21 12:53:35.502853666 (381h24m35.373936701s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502873894 (381h24m35.373956929s) [seg_pos={0 0}, upload=334718]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:53:35.502965338 (381h24m35.374048383s) [upload=334718]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:53:35.502970939 (381h24m35.374053984s) [upload=334718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:53:35.794077451 (381h24m35.665160486s) [upload=334718]: Done uploading segments... Dec 21 12:53:35.794103912 (381h24m35.665186957s) [upload=334718]: Upload finished. err: <nil> Duration: 5.894223753s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.....X.X.X.....X..X.....X..X....X........X..................] duration: 5.967959108s trace: https://jaeger.dev.storj.io/trace/5254881e4ca2c0fe FAIL (2025-12-21 13:06:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, Piece ID: YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A) context deadline exceeded; piecestore: (Node ID: 1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, Piece ID: I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ) context deadline exceeded; piecestore: (Node ID: 12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, Piece ID: 7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ) context deadline exceeded; piecestore: (Node ID: 17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, Piece ID: 36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q) context deadline exceeded ======LOGS====== ents... Dec 21 12:53:32.080943203 (381h24m31.952026248s) [upload=334718]: Got next segment. Inline: false Dec 21 12:53:32.080975975 (381h24m31.95205901s) [upload=334718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:53:32.080980384 (381h24m31.952063419s) [upload=334718]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:53:32.328092505 (381h24m32.19917554s) [seg_pos={0 0}, upload=334718]: Begin upload segment... Dec 21 12:53:32.328182436 (381h24m32.199265471s) [seg_pos={0 0}, upload=334718]: Done begin upload segment. Dec 21 12:53:32.328220579 (381h24m32.199303624s) [noise=true , address=145.79.227.48:4302 , piece_id=GIFTHYKLR2K3DJLHB4VZ6T7NMYSA5TRAKPTEEJ53UVMQTYROMECA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.331668685 (381h24m32.20275173s) [noise=true , address=145.79.227.52:4445 , piece_id=SJDQEFSKBN6ERSH2WJUKMY4TEKA2XJQHDU7YBGTZWB7OQ52WE3KA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.332500137 (381h24m32.203583172s) [noise=true , address=145.79.227.47:4267 , piece_id=YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.333277057 (381h24m32.204360092s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.334224581 (381h24m32.205307617s) [noise=true , address=145.79.227.42:4121 , piece_id=PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.334844001 (381h24m32.205927046s) [noise=true , address=45.91.116.6:4974 , piece_id=ZX3KN4IETHLW3ABY6J63I35OMTQC2ZMFKO3GBPU4PY4HDS6OA22A, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.335828375 (381h24m32.20691142s) [noise=true , address=145.79.227.29:4943 , piece_id=DZZBLVTCTKGAHIFCHYNS5JLXQD2IBSP6XELBRGWS4QUBPYFMH7SA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.336610345 (381h24m32.20769338s) [noise=true , address=145.79.227.49:4338 , piece_id=I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.337469029 (381h24m32.208552074s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.338354977 (381h24m32.209438022s) [noise=true , address=145.79.227.34:4907 , piece_id=7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.339032116 (381h24m32.210115161s) [noise=true , address=145.79.227.15:4875 , piece_id=J53SUZ5OSH7B6THTLOG6762X2VDONOODZQSPT4I3BKGOCZZV3HEA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.340261366 (381h24m32.211344401s) [noise=true , address=145.79.227.14:4839 , piece_id=RMPSFFROF52HBMWI7HYYOZRYPN3YM6CCUSZJQABGOCKQOEOMGPZA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.340406363 (381h24m32.211489398s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.341633339 (381h24m32.212716374s) [noise=true , address=145.79.227.33:4870 , piece_id=GO6YIYMOGWGKS3KHIKAUBTOYTJLENKH6M7HJWLHC3OTNMUA4WJXA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.342100158 (381h24m32.213183193s) [noise=true , address=145.79.227.41:4085 , piece_id=36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.343690196 (381h24m32.214773241s) [noise=true , address=145.79.227.32:4834 , piece_id=T6AGJLKTKW4MJTMXEBAW4YBF5BSWQRKHWYKLEU4Y5U37NAMMAJTQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:34.215235315 (381h24m34.08631835s) [upload=334718]: Next returned nil segment Dec 21 12:53:34.215255653 (381h24m34.086338689s) [upload=334718]: Waiting for error group managing segments... Dec 21 12:53:34.523622594 (381h24m34.394705629s) [noise=true , address=145.79.227.49:4338 , piece_id=I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.525892566 (381h24m34.396975601s) [noise=true , address=145.79.227.14:4839 , piece_id=RMPSFFROF52HBMWI7HYYOZRYPN3YM6CCUSZJQABGOCKQOEOMGPZA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.535347309 (381h24m34.406430344s) [noise=true , address=145.79.227.41:4085 , piece_id=36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.592006026 (381h24m34.463089061s) [noise=true , address=145.79.227.34:4907 , piece_id=7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.609875664 (381h24m34.480958709s) [noise=true , address=145.79.227.32:4834 , piece_id=T6AGJLKTKW4MJTMXEBAW4YBF5BSWQRKHWYKLEU4Y5U37NAMMAJTQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.622134556 (381h24m34.493217591s) [noise=true , address=145.79.227.15:4875 , piece_id=J53SUZ5OSH7B6THTLOG6762X2VDONOODZQSPT4I3BKGOCZZV3HEA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.755361442 (381h24m34.626444487s) [noise=true , address=145.79.227.29:4943 , piece_id=DZZBLVTCTKGAHIFCHYNS5JLXQD2IBSP6XELBRGWS4QUBPYFMH7SA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.876619305 (381h24m34.74770235s) [noise=true , address=145.79.227.33:4870 , piece_id=GO6YIYMOGWGKS3KHIKAUBTOYTJLENKH6M7HJWLHC3OTNMUA4WJXA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.899187028 (381h24m34.770270073s) [noise=true , address=145.79.227.52:4445 , piece_id=SJDQEFSKBN6ERSH2WJUKMY4TEKA2XJQHDU7YBGTZWB7OQ52WE3KA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.205676652 (381h24m35.076759698s) [noise=true , address=145.79.227.42:4121 , piece_id=PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.259291003 (381h24m35.130374048s) [noise=true , address=145.79.227.48:4302 , piece_id=GIFTHYKLR2K3DJLHB4VZ6T7NMYSA5TRAKPTEEJ53UVMQTYROMECA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.326071153 (381h24m35.197154188s) [noise=true , address=45.91.116.6:4974 , piece_id=ZX3KN4IETHLW3ABY6J63I35OMTQC2ZMFKO3GBPU4PY4HDS6OA22A, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.502232413 (381h24m35.373315448s) [noise=true , address=145.79.227.47:4267 , piece_id=YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.502253854 (381h24m35.373336919s) [seg_pos={0 0}, upload=334718]: Segment reached optimal threshold of 13 pieces. Dec 21 12:53:35.502568303 (381h24m35.373651338s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 21 12:53:35.502581327 (381h24m35.373664372s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502608810 (381h24m35.373691845s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection Dec 21 12:53:35.502619169 (381h24m35.373702214s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502846553 (381h24m35.373929588s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection Dec 21 12:53:35.502853666 (381h24m35.373936701s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502873894 (381h24m35.373956929s) [seg_pos={0 0}, upload=334718]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:53:35.502965338 (381h24m35.374048383s) [upload=334718]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:53:35.502970939 (381h24m35.374053984s) [upload=334718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:53:35.794077451 (381h24m35.665160486s) [upload=334718]: Done uploading segments... Dec 21 12:53:35.794103912 (381h24m35.665186957s) [upload=334718]: Upload finished. err: <nil> Duration: 5.967982072s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [......X.X.X.....X..X.....X..X....X........X.................] duration: 5.703613302s trace: https://jaeger.dev.storj.io/trace/5254881e4ca2c0fe FAIL (2025-12-21 13:06:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, Piece ID: YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A) context deadline exceeded; piecestore: (Node ID: 1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, Piece ID: I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ) context deadline exceeded; piecestore: (Node ID: 12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, Piece ID: 7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ) context deadline exceeded; piecestore: (Node ID: 17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, Piece ID: 36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q) context deadline exceeded ======LOGS====== ents... Dec 21 12:53:32.080943203 (381h24m31.952026248s) [upload=334718]: Got next segment. Inline: false Dec 21 12:53:32.080975975 (381h24m31.95205901s) [upload=334718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:53:32.080980384 (381h24m31.952063419s) [upload=334718]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:53:32.328092505 (381h24m32.19917554s) [seg_pos={0 0}, upload=334718]: Begin upload segment... Dec 21 12:53:32.328182436 (381h24m32.199265471s) [seg_pos={0 0}, upload=334718]: Done begin upload segment. Dec 21 12:53:32.328220579 (381h24m32.199303624s) [noise=true , address=145.79.227.48:4302 , piece_id=GIFTHYKLR2K3DJLHB4VZ6T7NMYSA5TRAKPTEEJ53UVMQTYROMECA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.331668685 (381h24m32.20275173s) [noise=true , address=145.79.227.52:4445 , piece_id=SJDQEFSKBN6ERSH2WJUKMY4TEKA2XJQHDU7YBGTZWB7OQ52WE3KA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.332500137 (381h24m32.203583172s) [noise=true , address=145.79.227.47:4267 , piece_id=YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.333277057 (381h24m32.204360092s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.334224581 (381h24m32.205307617s) [noise=true , address=145.79.227.42:4121 , piece_id=PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.334844001 (381h24m32.205927046s) [noise=true , address=45.91.116.6:4974 , piece_id=ZX3KN4IETHLW3ABY6J63I35OMTQC2ZMFKO3GBPU4PY4HDS6OA22A, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.335828375 (381h24m32.20691142s) [noise=true , address=145.79.227.29:4943 , piece_id=DZZBLVTCTKGAHIFCHYNS5JLXQD2IBSP6XELBRGWS4QUBPYFMH7SA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.336610345 (381h24m32.20769338s) [noise=true , address=145.79.227.49:4338 , piece_id=I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.337469029 (381h24m32.208552074s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.338354977 (381h24m32.209438022s) [noise=true , address=145.79.227.34:4907 , piece_id=7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.339032116 (381h24m32.210115161s) [noise=true , address=145.79.227.15:4875 , piece_id=J53SUZ5OSH7B6THTLOG6762X2VDONOODZQSPT4I3BKGOCZZV3HEA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.340261366 (381h24m32.211344401s) [noise=true , address=145.79.227.14:4839 , piece_id=RMPSFFROF52HBMWI7HYYOZRYPN3YM6CCUSZJQABGOCKQOEOMGPZA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.340406363 (381h24m32.211489398s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.341633339 (381h24m32.212716374s) [noise=true , address=145.79.227.33:4870 , piece_id=GO6YIYMOGWGKS3KHIKAUBTOYTJLENKH6M7HJWLHC3OTNMUA4WJXA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.342100158 (381h24m32.213183193s) [noise=true , address=145.79.227.41:4085 , piece_id=36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.343690196 (381h24m32.214773241s) [noise=true , address=145.79.227.32:4834 , piece_id=T6AGJLKTKW4MJTMXEBAW4YBF5BSWQRKHWYKLEU4Y5U37NAMMAJTQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:34.215235315 (381h24m34.08631835s) [upload=334718]: Next returned nil segment Dec 21 12:53:34.215255653 (381h24m34.086338689s) [upload=334718]: Waiting for error group managing segments... Dec 21 12:53:34.523622594 (381h24m34.394705629s) [noise=true , address=145.79.227.49:4338 , piece_id=I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.525892566 (381h24m34.396975601s) [noise=true , address=145.79.227.14:4839 , piece_id=RMPSFFROF52HBMWI7HYYOZRYPN3YM6CCUSZJQABGOCKQOEOMGPZA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.535347309 (381h24m34.406430344s) [noise=true , address=145.79.227.41:4085 , piece_id=36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.592006026 (381h24m34.463089061s) [noise=true , address=145.79.227.34:4907 , piece_id=7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.609875664 (381h24m34.480958709s) [noise=true , address=145.79.227.32:4834 , piece_id=T6AGJLKTKW4MJTMXEBAW4YBF5BSWQRKHWYKLEU4Y5U37NAMMAJTQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.622134556 (381h24m34.493217591s) [noise=true , address=145.79.227.15:4875 , piece_id=J53SUZ5OSH7B6THTLOG6762X2VDONOODZQSPT4I3BKGOCZZV3HEA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.755361442 (381h24m34.626444487s) [noise=true , address=145.79.227.29:4943 , piece_id=DZZBLVTCTKGAHIFCHYNS5JLXQD2IBSP6XELBRGWS4QUBPYFMH7SA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.876619305 (381h24m34.74770235s) [noise=true , address=145.79.227.33:4870 , piece_id=GO6YIYMOGWGKS3KHIKAUBTOYTJLENKH6M7HJWLHC3OTNMUA4WJXA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.899187028 (381h24m34.770270073s) [noise=true , address=145.79.227.52:4445 , piece_id=SJDQEFSKBN6ERSH2WJUKMY4TEKA2XJQHDU7YBGTZWB7OQ52WE3KA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.205676652 (381h24m35.076759698s) [noise=true , address=145.79.227.42:4121 , piece_id=PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.259291003 (381h24m35.130374048s) [noise=true , address=145.79.227.48:4302 , piece_id=GIFTHYKLR2K3DJLHB4VZ6T7NMYSA5TRAKPTEEJ53UVMQTYROMECA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.326071153 (381h24m35.197154188s) [noise=true , address=45.91.116.6:4974 , piece_id=ZX3KN4IETHLW3ABY6J63I35OMTQC2ZMFKO3GBPU4PY4HDS6OA22A, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.502232413 (381h24m35.373315448s) [noise=true , address=145.79.227.47:4267 , piece_id=YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.502253854 (381h24m35.373336919s) [seg_pos={0 0}, upload=334718]: Segment reached optimal threshold of 13 pieces. Dec 21 12:53:35.502568303 (381h24m35.373651338s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 21 12:53:35.502581327 (381h24m35.373664372s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502608810 (381h24m35.373691845s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection Dec 21 12:53:35.502619169 (381h24m35.373702214s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502846553 (381h24m35.373929588s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection Dec 21 12:53:35.502853666 (381h24m35.373936701s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502873894 (381h24m35.373956929s) [seg_pos={0 0}, upload=334718]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:53:35.502965338 (381h24m35.374048383s) [upload=334718]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:53:35.502970939 (381h24m35.374053984s) [upload=334718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:53:35.794077451 (381h24m35.665160486s) [upload=334718]: Done uploading segments... Dec 21 12:53:35.794103912 (381h24m35.665186957s) [upload=334718]: Upload finished. err: <nil> Duration: 5.703662716s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.......X.X.X.....X..X.....X..X....X........X................] duration: 6.070717883s trace: https://jaeger.dev.storj.io/trace/5254881e4ca2c0fe FAIL (2025-12-21 13:05:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, Piece ID: YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A) context deadline exceeded; piecestore: (Node ID: 1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, Piece ID: I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ) context deadline exceeded; piecestore: (Node ID: 12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, Piece ID: 7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ) context deadline exceeded; piecestore: (Node ID: 17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, Piece ID: 36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q) context deadline exceeded ======LOGS====== ents... Dec 21 12:53:32.080943203 (381h24m31.952026248s) [upload=334718]: Got next segment. Inline: false Dec 21 12:53:32.080975975 (381h24m31.95205901s) [upload=334718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:53:32.080980384 (381h24m31.952063419s) [upload=334718]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:53:32.328092505 (381h24m32.19917554s) [seg_pos={0 0}, upload=334718]: Begin upload segment... Dec 21 12:53:32.328182436 (381h24m32.199265471s) [seg_pos={0 0}, upload=334718]: Done begin upload segment. Dec 21 12:53:32.328220579 (381h24m32.199303624s) [noise=true , address=145.79.227.48:4302 , piece_id=GIFTHYKLR2K3DJLHB4VZ6T7NMYSA5TRAKPTEEJ53UVMQTYROMECA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.331668685 (381h24m32.20275173s) [noise=true , address=145.79.227.52:4445 , piece_id=SJDQEFSKBN6ERSH2WJUKMY4TEKA2XJQHDU7YBGTZWB7OQ52WE3KA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.332500137 (381h24m32.203583172s) [noise=true , address=145.79.227.47:4267 , piece_id=YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.333277057 (381h24m32.204360092s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.334224581 (381h24m32.205307617s) [noise=true , address=145.79.227.42:4121 , piece_id=PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.334844001 (381h24m32.205927046s) [noise=true , address=45.91.116.6:4974 , piece_id=ZX3KN4IETHLW3ABY6J63I35OMTQC2ZMFKO3GBPU4PY4HDS6OA22A, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.335828375 (381h24m32.20691142s) [noise=true , address=145.79.227.29:4943 , piece_id=DZZBLVTCTKGAHIFCHYNS5JLXQD2IBSP6XELBRGWS4QUBPYFMH7SA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.336610345 (381h24m32.20769338s) [noise=true , address=145.79.227.49:4338 , piece_id=I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.337469029 (381h24m32.208552074s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.338354977 (381h24m32.209438022s) [noise=true , address=145.79.227.34:4907 , piece_id=7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.339032116 (381h24m32.210115161s) [noise=true , address=145.79.227.15:4875 , piece_id=J53SUZ5OSH7B6THTLOG6762X2VDONOODZQSPT4I3BKGOCZZV3HEA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.340261366 (381h24m32.211344401s) [noise=true , address=145.79.227.14:4839 , piece_id=RMPSFFROF52HBMWI7HYYOZRYPN3YM6CCUSZJQABGOCKQOEOMGPZA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.340406363 (381h24m32.211489398s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.341633339 (381h24m32.212716374s) [noise=true , address=145.79.227.33:4870 , piece_id=GO6YIYMOGWGKS3KHIKAUBTOYTJLENKH6M7HJWLHC3OTNMUA4WJXA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.342100158 (381h24m32.213183193s) [noise=true , address=145.79.227.41:4085 , piece_id=36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.343690196 (381h24m32.214773241s) [noise=true , address=145.79.227.32:4834 , piece_id=T6AGJLKTKW4MJTMXEBAW4YBF5BSWQRKHWYKLEU4Y5U37NAMMAJTQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:34.215235315 (381h24m34.08631835s) [upload=334718]: Next returned nil segment Dec 21 12:53:34.215255653 (381h24m34.086338689s) [upload=334718]: Waiting for error group managing segments... Dec 21 12:53:34.523622594 (381h24m34.394705629s) [noise=true , address=145.79.227.49:4338 , piece_id=I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.525892566 (381h24m34.396975601s) [noise=true , address=145.79.227.14:4839 , piece_id=RMPSFFROF52HBMWI7HYYOZRYPN3YM6CCUSZJQABGOCKQOEOMGPZA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.535347309 (381h24m34.406430344s) [noise=true , address=145.79.227.41:4085 , piece_id=36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.592006026 (381h24m34.463089061s) [noise=true , address=145.79.227.34:4907 , piece_id=7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.609875664 (381h24m34.480958709s) [noise=true , address=145.79.227.32:4834 , piece_id=T6AGJLKTKW4MJTMXEBAW4YBF5BSWQRKHWYKLEU4Y5U37NAMMAJTQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.622134556 (381h24m34.493217591s) [noise=true , address=145.79.227.15:4875 , piece_id=J53SUZ5OSH7B6THTLOG6762X2VDONOODZQSPT4I3BKGOCZZV3HEA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.755361442 (381h24m34.626444487s) [noise=true , address=145.79.227.29:4943 , piece_id=DZZBLVTCTKGAHIFCHYNS5JLXQD2IBSP6XELBRGWS4QUBPYFMH7SA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.876619305 (381h24m34.74770235s) [noise=true , address=145.79.227.33:4870 , piece_id=GO6YIYMOGWGKS3KHIKAUBTOYTJLENKH6M7HJWLHC3OTNMUA4WJXA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.899187028 (381h24m34.770270073s) [noise=true , address=145.79.227.52:4445 , piece_id=SJDQEFSKBN6ERSH2WJUKMY4TEKA2XJQHDU7YBGTZWB7OQ52WE3KA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.205676652 (381h24m35.076759698s) [noise=true , address=145.79.227.42:4121 , piece_id=PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.259291003 (381h24m35.130374048s) [noise=true , address=145.79.227.48:4302 , piece_id=GIFTHYKLR2K3DJLHB4VZ6T7NMYSA5TRAKPTEEJ53UVMQTYROMECA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.326071153 (381h24m35.197154188s) [noise=true , address=45.91.116.6:4974 , piece_id=ZX3KN4IETHLW3ABY6J63I35OMTQC2ZMFKO3GBPU4PY4HDS6OA22A, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.502232413 (381h24m35.373315448s) [noise=true , address=145.79.227.47:4267 , piece_id=YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.502253854 (381h24m35.373336919s) [seg_pos={0 0}, upload=334718]: Segment reached optimal threshold of 13 pieces. Dec 21 12:53:35.502568303 (381h24m35.373651338s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 21 12:53:35.502581327 (381h24m35.373664372s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502608810 (381h24m35.373691845s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection Dec 21 12:53:35.502619169 (381h24m35.373702214s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502846553 (381h24m35.373929588s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection Dec 21 12:53:35.502853666 (381h24m35.373936701s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502873894 (381h24m35.373956929s) [seg_pos={0 0}, upload=334718]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:53:35.502965338 (381h24m35.374048383s) [upload=334718]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:53:35.502970939 (381h24m35.374053984s) [upload=334718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:53:35.794077451 (381h24m35.665160486s) [upload=334718]: Done uploading segments... Dec 21 12:53:35.794103912 (381h24m35.665186957s) [upload=334718]: Upload finished. err: <nil> Duration: 6.070745917s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [........X.X.X.....X..X.....X..X....X........X...............] duration: 6.221807893s trace: https://jaeger.dev.storj.io/trace/5254881e4ca2c0fe FAIL (2025-12-21 13:05:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, Piece ID: YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A) context deadline exceeded; piecestore: (Node ID: 1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, Piece ID: I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ) context deadline exceeded; piecestore: (Node ID: 12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, Piece ID: 7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ) context deadline exceeded; piecestore: (Node ID: 17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, Piece ID: 36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q) context deadline exceeded ======LOGS====== ents... Dec 21 12:53:32.080943203 (381h24m31.952026248s) [upload=334718]: Got next segment. Inline: false Dec 21 12:53:32.080975975 (381h24m31.95205901s) [upload=334718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:53:32.080980384 (381h24m31.952063419s) [upload=334718]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:53:32.328092505 (381h24m32.19917554s) [seg_pos={0 0}, upload=334718]: Begin upload segment... Dec 21 12:53:32.328182436 (381h24m32.199265471s) [seg_pos={0 0}, upload=334718]: Done begin upload segment. Dec 21 12:53:32.328220579 (381h24m32.199303624s) [noise=true , address=145.79.227.48:4302 , piece_id=GIFTHYKLR2K3DJLHB4VZ6T7NMYSA5TRAKPTEEJ53UVMQTYROMECA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.331668685 (381h24m32.20275173s) [noise=true , address=145.79.227.52:4445 , piece_id=SJDQEFSKBN6ERSH2WJUKMY4TEKA2XJQHDU7YBGTZWB7OQ52WE3KA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.332500137 (381h24m32.203583172s) [noise=true , address=145.79.227.47:4267 , piece_id=YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.333277057 (381h24m32.204360092s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.334224581 (381h24m32.205307617s) [noise=true , address=145.79.227.42:4121 , piece_id=PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.334844001 (381h24m32.205927046s) [noise=true , address=45.91.116.6:4974 , piece_id=ZX3KN4IETHLW3ABY6J63I35OMTQC2ZMFKO3GBPU4PY4HDS6OA22A, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.335828375 (381h24m32.20691142s) [noise=true , address=145.79.227.29:4943 , piece_id=DZZBLVTCTKGAHIFCHYNS5JLXQD2IBSP6XELBRGWS4QUBPYFMH7SA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.336610345 (381h24m32.20769338s) [noise=true , address=145.79.227.49:4338 , piece_id=I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.337469029 (381h24m32.208552074s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.338354977 (381h24m32.209438022s) [noise=true , address=145.79.227.34:4907 , piece_id=7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.339032116 (381h24m32.210115161s) [noise=true , address=145.79.227.15:4875 , piece_id=J53SUZ5OSH7B6THTLOG6762X2VDONOODZQSPT4I3BKGOCZZV3HEA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.340261366 (381h24m32.211344401s) [noise=true , address=145.79.227.14:4839 , piece_id=RMPSFFROF52HBMWI7HYYOZRYPN3YM6CCUSZJQABGOCKQOEOMGPZA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.340406363 (381h24m32.211489398s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.341633339 (381h24m32.212716374s) [noise=true , address=145.79.227.33:4870 , piece_id=GO6YIYMOGWGKS3KHIKAUBTOYTJLENKH6M7HJWLHC3OTNMUA4WJXA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.342100158 (381h24m32.213183193s) [noise=true , address=145.79.227.41:4085 , piece_id=36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.343690196 (381h24m32.214773241s) [noise=true , address=145.79.227.32:4834 , piece_id=T6AGJLKTKW4MJTMXEBAW4YBF5BSWQRKHWYKLEU4Y5U37NAMMAJTQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:34.215235315 (381h24m34.08631835s) [upload=334718]: Next returned nil segment Dec 21 12:53:34.215255653 (381h24m34.086338689s) [upload=334718]: Waiting for error group managing segments... Dec 21 12:53:34.523622594 (381h24m34.394705629s) [noise=true , address=145.79.227.49:4338 , piece_id=I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.525892566 (381h24m34.396975601s) [noise=true , address=145.79.227.14:4839 , piece_id=RMPSFFROF52HBMWI7HYYOZRYPN3YM6CCUSZJQABGOCKQOEOMGPZA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.535347309 (381h24m34.406430344s) [noise=true , address=145.79.227.41:4085 , piece_id=36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.592006026 (381h24m34.463089061s) [noise=true , address=145.79.227.34:4907 , piece_id=7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.609875664 (381h24m34.480958709s) [noise=true , address=145.79.227.32:4834 , piece_id=T6AGJLKTKW4MJTMXEBAW4YBF5BSWQRKHWYKLEU4Y5U37NAMMAJTQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.622134556 (381h24m34.493217591s) [noise=true , address=145.79.227.15:4875 , piece_id=J53SUZ5OSH7B6THTLOG6762X2VDONOODZQSPT4I3BKGOCZZV3HEA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.755361442 (381h24m34.626444487s) [noise=true , address=145.79.227.29:4943 , piece_id=DZZBLVTCTKGAHIFCHYNS5JLXQD2IBSP6XELBRGWS4QUBPYFMH7SA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.876619305 (381h24m34.74770235s) [noise=true , address=145.79.227.33:4870 , piece_id=GO6YIYMOGWGKS3KHIKAUBTOYTJLENKH6M7HJWLHC3OTNMUA4WJXA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.899187028 (381h24m34.770270073s) [noise=true , address=145.79.227.52:4445 , piece_id=SJDQEFSKBN6ERSH2WJUKMY4TEKA2XJQHDU7YBGTZWB7OQ52WE3KA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.205676652 (381h24m35.076759698s) [noise=true , address=145.79.227.42:4121 , piece_id=PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.259291003 (381h24m35.130374048s) [noise=true , address=145.79.227.48:4302 , piece_id=GIFTHYKLR2K3DJLHB4VZ6T7NMYSA5TRAKPTEEJ53UVMQTYROMECA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.326071153 (381h24m35.197154188s) [noise=true , address=45.91.116.6:4974 , piece_id=ZX3KN4IETHLW3ABY6J63I35OMTQC2ZMFKO3GBPU4PY4HDS6OA22A, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.502232413 (381h24m35.373315448s) [noise=true , address=145.79.227.47:4267 , piece_id=YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.502253854 (381h24m35.373336919s) [seg_pos={0 0}, upload=334718]: Segment reached optimal threshold of 13 pieces. Dec 21 12:53:35.502568303 (381h24m35.373651338s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 21 12:53:35.502581327 (381h24m35.373664372s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502608810 (381h24m35.373691845s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection Dec 21 12:53:35.502619169 (381h24m35.373702214s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502846553 (381h24m35.373929588s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection Dec 21 12:53:35.502853666 (381h24m35.373936701s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502873894 (381h24m35.373956929s) [seg_pos={0 0}, upload=334718]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:53:35.502965338 (381h24m35.374048383s) [upload=334718]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:53:35.502970939 (381h24m35.374053984s) [upload=334718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:53:35.794077451 (381h24m35.665160486s) [upload=334718]: Done uploading segments... Dec 21 12:53:35.794103912 (381h24m35.665186957s) [upload=334718]: Upload finished. err: <nil> Duration: 6.221834554s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.........X.X.X.....X..X.....X..X....X........X..............] duration: 5.650729668s trace: https://jaeger.dev.storj.io/trace/5254881e4ca2c0fe FAIL (2025-12-21 13:04:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, Piece ID: YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A) context deadline exceeded; piecestore: (Node ID: 1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, Piece ID: I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ) context deadline exceeded; piecestore: (Node ID: 12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, Piece ID: 7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ) context deadline exceeded; piecestore: (Node ID: 17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, Piece ID: 36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q) context deadline exceeded ======LOGS====== ents... Dec 21 12:53:32.080943203 (381h24m31.952026248s) [upload=334718]: Got next segment. Inline: false Dec 21 12:53:32.080975975 (381h24m31.95205901s) [upload=334718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:53:32.080980384 (381h24m31.952063419s) [upload=334718]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:53:32.328092505 (381h24m32.19917554s) [seg_pos={0 0}, upload=334718]: Begin upload segment... Dec 21 12:53:32.328182436 (381h24m32.199265471s) [seg_pos={0 0}, upload=334718]: Done begin upload segment. Dec 21 12:53:32.328220579 (381h24m32.199303624s) [noise=true , address=145.79.227.48:4302 , piece_id=GIFTHYKLR2K3DJLHB4VZ6T7NMYSA5TRAKPTEEJ53UVMQTYROMECA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.331668685 (381h24m32.20275173s) [noise=true , address=145.79.227.52:4445 , piece_id=SJDQEFSKBN6ERSH2WJUKMY4TEKA2XJQHDU7YBGTZWB7OQ52WE3KA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.332500137 (381h24m32.203583172s) [noise=true , address=145.79.227.47:4267 , piece_id=YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.333277057 (381h24m32.204360092s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.334224581 (381h24m32.205307617s) [noise=true , address=145.79.227.42:4121 , piece_id=PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.334844001 (381h24m32.205927046s) [noise=true , address=45.91.116.6:4974 , piece_id=ZX3KN4IETHLW3ABY6J63I35OMTQC2ZMFKO3GBPU4PY4HDS6OA22A, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.335828375 (381h24m32.20691142s) [noise=true , address=145.79.227.29:4943 , piece_id=DZZBLVTCTKGAHIFCHYNS5JLXQD2IBSP6XELBRGWS4QUBPYFMH7SA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.336610345 (381h24m32.20769338s) [noise=true , address=145.79.227.49:4338 , piece_id=I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.337469029 (381h24m32.208552074s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.338354977 (381h24m32.209438022s) [noise=true , address=145.79.227.34:4907 , piece_id=7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.339032116 (381h24m32.210115161s) [noise=true , address=145.79.227.15:4875 , piece_id=J53SUZ5OSH7B6THTLOG6762X2VDONOODZQSPT4I3BKGOCZZV3HEA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.340261366 (381h24m32.211344401s) [noise=true , address=145.79.227.14:4839 , piece_id=RMPSFFROF52HBMWI7HYYOZRYPN3YM6CCUSZJQABGOCKQOEOMGPZA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.340406363 (381h24m32.211489398s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.341633339 (381h24m32.212716374s) [noise=true , address=145.79.227.33:4870 , piece_id=GO6YIYMOGWGKS3KHIKAUBTOYTJLENKH6M7HJWLHC3OTNMUA4WJXA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.342100158 (381h24m32.213183193s) [noise=true , address=145.79.227.41:4085 , piece_id=36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.343690196 (381h24m32.214773241s) [noise=true , address=145.79.227.32:4834 , piece_id=T6AGJLKTKW4MJTMXEBAW4YBF5BSWQRKHWYKLEU4Y5U37NAMMAJTQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:34.215235315 (381h24m34.08631835s) [upload=334718]: Next returned nil segment Dec 21 12:53:34.215255653 (381h24m34.086338689s) [upload=334718]: Waiting for error group managing segments... Dec 21 12:53:34.523622594 (381h24m34.394705629s) [noise=true , address=145.79.227.49:4338 , piece_id=I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.525892566 (381h24m34.396975601s) [noise=true , address=145.79.227.14:4839 , piece_id=RMPSFFROF52HBMWI7HYYOZRYPN3YM6CCUSZJQABGOCKQOEOMGPZA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.535347309 (381h24m34.406430344s) [noise=true , address=145.79.227.41:4085 , piece_id=36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.592006026 (381h24m34.463089061s) [noise=true , address=145.79.227.34:4907 , piece_id=7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.609875664 (381h24m34.480958709s) [noise=true , address=145.79.227.32:4834 , piece_id=T6AGJLKTKW4MJTMXEBAW4YBF5BSWQRKHWYKLEU4Y5U37NAMMAJTQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.622134556 (381h24m34.493217591s) [noise=true , address=145.79.227.15:4875 , piece_id=J53SUZ5OSH7B6THTLOG6762X2VDONOODZQSPT4I3BKGOCZZV3HEA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.755361442 (381h24m34.626444487s) [noise=true , address=145.79.227.29:4943 , piece_id=DZZBLVTCTKGAHIFCHYNS5JLXQD2IBSP6XELBRGWS4QUBPYFMH7SA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.876619305 (381h24m34.74770235s) [noise=true , address=145.79.227.33:4870 , piece_id=GO6YIYMOGWGKS3KHIKAUBTOYTJLENKH6M7HJWLHC3OTNMUA4WJXA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.899187028 (381h24m34.770270073s) [noise=true , address=145.79.227.52:4445 , piece_id=SJDQEFSKBN6ERSH2WJUKMY4TEKA2XJQHDU7YBGTZWB7OQ52WE3KA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.205676652 (381h24m35.076759698s) [noise=true , address=145.79.227.42:4121 , piece_id=PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.259291003 (381h24m35.130374048s) [noise=true , address=145.79.227.48:4302 , piece_id=GIFTHYKLR2K3DJLHB4VZ6T7NMYSA5TRAKPTEEJ53UVMQTYROMECA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.326071153 (381h24m35.197154188s) [noise=true , address=45.91.116.6:4974 , piece_id=ZX3KN4IETHLW3ABY6J63I35OMTQC2ZMFKO3GBPU4PY4HDS6OA22A, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.502232413 (381h24m35.373315448s) [noise=true , address=145.79.227.47:4267 , piece_id=YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.502253854 (381h24m35.373336919s) [seg_pos={0 0}, upload=334718]: Segment reached optimal threshold of 13 pieces. Dec 21 12:53:35.502568303 (381h24m35.373651338s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 21 12:53:35.502581327 (381h24m35.373664372s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502608810 (381h24m35.373691845s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection Dec 21 12:53:35.502619169 (381h24m35.373702214s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502846553 (381h24m35.373929588s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection Dec 21 12:53:35.502853666 (381h24m35.373936701s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502873894 (381h24m35.373956929s) [seg_pos={0 0}, upload=334718]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:53:35.502965338 (381h24m35.374048383s) [upload=334718]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:53:35.502970939 (381h24m35.374053984s) [upload=334718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:53:35.794077451 (381h24m35.665160486s) [upload=334718]: Done uploading segments... Dec 21 12:53:35.794103912 (381h24m35.665186957s) [upload=334718]: Upload finished. err: <nil> Duration: 5.650762421s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X.........X.X.X.....X..X.....X..X....X........X.............] duration: 6.006959137s trace: https://jaeger.dev.storj.io/trace/5254881e4ca2c0fe FAIL (2025-12-21 13:03:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, Piece ID: YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A) context deadline exceeded; piecestore: (Node ID: 1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, Piece ID: I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ) context deadline exceeded; piecestore: (Node ID: 12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, Piece ID: 7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ) context deadline exceeded; piecestore: (Node ID: 17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, Piece ID: 36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q) context deadline exceeded ======LOGS====== ents... Dec 21 12:53:32.080943203 (381h24m31.952026248s) [upload=334718]: Got next segment. Inline: false Dec 21 12:53:32.080975975 (381h24m31.95205901s) [upload=334718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:53:32.080980384 (381h24m31.952063419s) [upload=334718]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:53:32.328092505 (381h24m32.19917554s) [seg_pos={0 0}, upload=334718]: Begin upload segment... Dec 21 12:53:32.328182436 (381h24m32.199265471s) [seg_pos={0 0}, upload=334718]: Done begin upload segment. Dec 21 12:53:32.328220579 (381h24m32.199303624s) [noise=true , address=145.79.227.48:4302 , piece_id=GIFTHYKLR2K3DJLHB4VZ6T7NMYSA5TRAKPTEEJ53UVMQTYROMECA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.331668685 (381h24m32.20275173s) [noise=true , address=145.79.227.52:4445 , piece_id=SJDQEFSKBN6ERSH2WJUKMY4TEKA2XJQHDU7YBGTZWB7OQ52WE3KA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.332500137 (381h24m32.203583172s) [noise=true , address=145.79.227.47:4267 , piece_id=YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.333277057 (381h24m32.204360092s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.334224581 (381h24m32.205307617s) [noise=true , address=145.79.227.42:4121 , piece_id=PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.334844001 (381h24m32.205927046s) [noise=true , address=45.91.116.6:4974 , piece_id=ZX3KN4IETHLW3ABY6J63I35OMTQC2ZMFKO3GBPU4PY4HDS6OA22A, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.335828375 (381h24m32.20691142s) [noise=true , address=145.79.227.29:4943 , piece_id=DZZBLVTCTKGAHIFCHYNS5JLXQD2IBSP6XELBRGWS4QUBPYFMH7SA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.336610345 (381h24m32.20769338s) [noise=true , address=145.79.227.49:4338 , piece_id=I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.337469029 (381h24m32.208552074s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.338354977 (381h24m32.209438022s) [noise=true , address=145.79.227.34:4907 , piece_id=7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.339032116 (381h24m32.210115161s) [noise=true , address=145.79.227.15:4875 , piece_id=J53SUZ5OSH7B6THTLOG6762X2VDONOODZQSPT4I3BKGOCZZV3HEA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.340261366 (381h24m32.211344401s) [noise=true , address=145.79.227.14:4839 , piece_id=RMPSFFROF52HBMWI7HYYOZRYPN3YM6CCUSZJQABGOCKQOEOMGPZA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.340406363 (381h24m32.211489398s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.341633339 (381h24m32.212716374s) [noise=true , address=145.79.227.33:4870 , piece_id=GO6YIYMOGWGKS3KHIKAUBTOYTJLENKH6M7HJWLHC3OTNMUA4WJXA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.342100158 (381h24m32.213183193s) [noise=true , address=145.79.227.41:4085 , piece_id=36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.343690196 (381h24m32.214773241s) [noise=true , address=145.79.227.32:4834 , piece_id=T6AGJLKTKW4MJTMXEBAW4YBF5BSWQRKHWYKLEU4Y5U37NAMMAJTQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:34.215235315 (381h24m34.08631835s) [upload=334718]: Next returned nil segment Dec 21 12:53:34.215255653 (381h24m34.086338689s) [upload=334718]: Waiting for error group managing segments... Dec 21 12:53:34.523622594 (381h24m34.394705629s) [noise=true , address=145.79.227.49:4338 , piece_id=I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.525892566 (381h24m34.396975601s) [noise=true , address=145.79.227.14:4839 , piece_id=RMPSFFROF52HBMWI7HYYOZRYPN3YM6CCUSZJQABGOCKQOEOMGPZA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.535347309 (381h24m34.406430344s) [noise=true , address=145.79.227.41:4085 , piece_id=36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.592006026 (381h24m34.463089061s) [noise=true , address=145.79.227.34:4907 , piece_id=7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.609875664 (381h24m34.480958709s) [noise=true , address=145.79.227.32:4834 , piece_id=T6AGJLKTKW4MJTMXEBAW4YBF5BSWQRKHWYKLEU4Y5U37NAMMAJTQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.622134556 (381h24m34.493217591s) [noise=true , address=145.79.227.15:4875 , piece_id=J53SUZ5OSH7B6THTLOG6762X2VDONOODZQSPT4I3BKGOCZZV3HEA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.755361442 (381h24m34.626444487s) [noise=true , address=145.79.227.29:4943 , piece_id=DZZBLVTCTKGAHIFCHYNS5JLXQD2IBSP6XELBRGWS4QUBPYFMH7SA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.876619305 (381h24m34.74770235s) [noise=true , address=145.79.227.33:4870 , piece_id=GO6YIYMOGWGKS3KHIKAUBTOYTJLENKH6M7HJWLHC3OTNMUA4WJXA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.899187028 (381h24m34.770270073s) [noise=true , address=145.79.227.52:4445 , piece_id=SJDQEFSKBN6ERSH2WJUKMY4TEKA2XJQHDU7YBGTZWB7OQ52WE3KA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.205676652 (381h24m35.076759698s) [noise=true , address=145.79.227.42:4121 , piece_id=PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.259291003 (381h24m35.130374048s) [noise=true , address=145.79.227.48:4302 , piece_id=GIFTHYKLR2K3DJLHB4VZ6T7NMYSA5TRAKPTEEJ53UVMQTYROMECA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.326071153 (381h24m35.197154188s) [noise=true , address=45.91.116.6:4974 , piece_id=ZX3KN4IETHLW3ABY6J63I35OMTQC2ZMFKO3GBPU4PY4HDS6OA22A, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.502232413 (381h24m35.373315448s) [noise=true , address=145.79.227.47:4267 , piece_id=YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.502253854 (381h24m35.373336919s) [seg_pos={0 0}, upload=334718]: Segment reached optimal threshold of 13 pieces. Dec 21 12:53:35.502568303 (381h24m35.373651338s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 21 12:53:35.502581327 (381h24m35.373664372s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502608810 (381h24m35.373691845s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection Dec 21 12:53:35.502619169 (381h24m35.373702214s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502846553 (381h24m35.373929588s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection Dec 21 12:53:35.502853666 (381h24m35.373936701s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502873894 (381h24m35.373956929s) [seg_pos={0 0}, upload=334718]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:53:35.502965338 (381h24m35.374048383s) [upload=334718]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:53:35.502970939 (381h24m35.374053984s) [upload=334718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:53:35.794077451 (381h24m35.665160486s) [upload=334718]: Done uploading segments... Dec 21 12:53:35.794103912 (381h24m35.665186957s) [upload=334718]: Upload finished. err: <nil> Duration: 6.006984886s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.X.........X.X.X.....X..X.....X..X....X........X............] duration: 5.905598879s trace: https://jaeger.dev.storj.io/trace/5254881e4ca2c0fe FAIL (2025-12-21 13:02:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, Piece ID: YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A) context deadline exceeded; piecestore: (Node ID: 1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, Piece ID: I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ) context deadline exceeded; piecestore: (Node ID: 12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, Piece ID: 7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ) context deadline exceeded; piecestore: (Node ID: 17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, Piece ID: 36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q) context deadline exceeded ======LOGS====== ents... Dec 21 12:53:32.080943203 (381h24m31.952026248s) [upload=334718]: Got next segment. Inline: false Dec 21 12:53:32.080975975 (381h24m31.95205901s) [upload=334718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:53:32.080980384 (381h24m31.952063419s) [upload=334718]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:53:32.328092505 (381h24m32.19917554s) [seg_pos={0 0}, upload=334718]: Begin upload segment... Dec 21 12:53:32.328182436 (381h24m32.199265471s) [seg_pos={0 0}, upload=334718]: Done begin upload segment. Dec 21 12:53:32.328220579 (381h24m32.199303624s) [noise=true , address=145.79.227.48:4302 , piece_id=GIFTHYKLR2K3DJLHB4VZ6T7NMYSA5TRAKPTEEJ53UVMQTYROMECA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.331668685 (381h24m32.20275173s) [noise=true , address=145.79.227.52:4445 , piece_id=SJDQEFSKBN6ERSH2WJUKMY4TEKA2XJQHDU7YBGTZWB7OQ52WE3KA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.332500137 (381h24m32.203583172s) [noise=true , address=145.79.227.47:4267 , piece_id=YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.333277057 (381h24m32.204360092s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.334224581 (381h24m32.205307617s) [noise=true , address=145.79.227.42:4121 , piece_id=PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.334844001 (381h24m32.205927046s) [noise=true , address=45.91.116.6:4974 , piece_id=ZX3KN4IETHLW3ABY6J63I35OMTQC2ZMFKO3GBPU4PY4HDS6OA22A, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.335828375 (381h24m32.20691142s) [noise=true , address=145.79.227.29:4943 , piece_id=DZZBLVTCTKGAHIFCHYNS5JLXQD2IBSP6XELBRGWS4QUBPYFMH7SA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.336610345 (381h24m32.20769338s) [noise=true , address=145.79.227.49:4338 , piece_id=I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.337469029 (381h24m32.208552074s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.338354977 (381h24m32.209438022s) [noise=true , address=145.79.227.34:4907 , piece_id=7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.339032116 (381h24m32.210115161s) [noise=true , address=145.79.227.15:4875 , piece_id=J53SUZ5OSH7B6THTLOG6762X2VDONOODZQSPT4I3BKGOCZZV3HEA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.340261366 (381h24m32.211344401s) [noise=true , address=145.79.227.14:4839 , piece_id=RMPSFFROF52HBMWI7HYYOZRYPN3YM6CCUSZJQABGOCKQOEOMGPZA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.340406363 (381h24m32.211489398s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.341633339 (381h24m32.212716374s) [noise=true , address=145.79.227.33:4870 , piece_id=GO6YIYMOGWGKS3KHIKAUBTOYTJLENKH6M7HJWLHC3OTNMUA4WJXA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.342100158 (381h24m32.213183193s) [noise=true , address=145.79.227.41:4085 , piece_id=36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.343690196 (381h24m32.214773241s) [noise=true , address=145.79.227.32:4834 , piece_id=T6AGJLKTKW4MJTMXEBAW4YBF5BSWQRKHWYKLEU4Y5U37NAMMAJTQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:34.215235315 (381h24m34.08631835s) [upload=334718]: Next returned nil segment Dec 21 12:53:34.215255653 (381h24m34.086338689s) [upload=334718]: Waiting for error group managing segments... Dec 21 12:53:34.523622594 (381h24m34.394705629s) [noise=true , address=145.79.227.49:4338 , piece_id=I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.525892566 (381h24m34.396975601s) [noise=true , address=145.79.227.14:4839 , piece_id=RMPSFFROF52HBMWI7HYYOZRYPN3YM6CCUSZJQABGOCKQOEOMGPZA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.535347309 (381h24m34.406430344s) [noise=true , address=145.79.227.41:4085 , piece_id=36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.592006026 (381h24m34.463089061s) [noise=true , address=145.79.227.34:4907 , piece_id=7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.609875664 (381h24m34.480958709s) [noise=true , address=145.79.227.32:4834 , piece_id=T6AGJLKTKW4MJTMXEBAW4YBF5BSWQRKHWYKLEU4Y5U37NAMMAJTQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.622134556 (381h24m34.493217591s) [noise=true , address=145.79.227.15:4875 , piece_id=J53SUZ5OSH7B6THTLOG6762X2VDONOODZQSPT4I3BKGOCZZV3HEA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.755361442 (381h24m34.626444487s) [noise=true , address=145.79.227.29:4943 , piece_id=DZZBLVTCTKGAHIFCHYNS5JLXQD2IBSP6XELBRGWS4QUBPYFMH7SA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.876619305 (381h24m34.74770235s) [noise=true , address=145.79.227.33:4870 , piece_id=GO6YIYMOGWGKS3KHIKAUBTOYTJLENKH6M7HJWLHC3OTNMUA4WJXA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.899187028 (381h24m34.770270073s) [noise=true , address=145.79.227.52:4445 , piece_id=SJDQEFSKBN6ERSH2WJUKMY4TEKA2XJQHDU7YBGTZWB7OQ52WE3KA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.205676652 (381h24m35.076759698s) [noise=true , address=145.79.227.42:4121 , piece_id=PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.259291003 (381h24m35.130374048s) [noise=true , address=145.79.227.48:4302 , piece_id=GIFTHYKLR2K3DJLHB4VZ6T7NMYSA5TRAKPTEEJ53UVMQTYROMECA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.326071153 (381h24m35.197154188s) [noise=true , address=45.91.116.6:4974 , piece_id=ZX3KN4IETHLW3ABY6J63I35OMTQC2ZMFKO3GBPU4PY4HDS6OA22A, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.502232413 (381h24m35.373315448s) [noise=true , address=145.79.227.47:4267 , piece_id=YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.502253854 (381h24m35.373336919s) [seg_pos={0 0}, upload=334718]: Segment reached optimal threshold of 13 pieces. Dec 21 12:53:35.502568303 (381h24m35.373651338s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 21 12:53:35.502581327 (381h24m35.373664372s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502608810 (381h24m35.373691845s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection Dec 21 12:53:35.502619169 (381h24m35.373702214s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502846553 (381h24m35.373929588s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection Dec 21 12:53:35.502853666 (381h24m35.373936701s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502873894 (381h24m35.373956929s) [seg_pos={0 0}, upload=334718]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:53:35.502965338 (381h24m35.374048383s) [upload=334718]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:53:35.502970939 (381h24m35.374053984s) [upload=334718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:53:35.794077451 (381h24m35.665160486s) [upload=334718]: Done uploading segments... Dec 21 12:53:35.794103912 (381h24m35.665186957s) [upload=334718]: Upload finished. err: <nil> Duration: 5.905627723s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..X.........X.X.X.....X..X.....X..X....X........X...........] duration: 6.02868782s trace: https://jaeger.dev.storj.io/trace/5254881e4ca2c0fe FAIL (2025-12-21 13:01:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, Piece ID: YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A) context deadline exceeded; piecestore: (Node ID: 1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, Piece ID: I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ) context deadline exceeded; piecestore: (Node ID: 12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, Piece ID: 7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ) context deadline exceeded; piecestore: (Node ID: 17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, Piece ID: 36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q) context deadline exceeded ======LOGS====== ents... Dec 21 12:53:32.080943203 (381h24m31.952026248s) [upload=334718]: Got next segment. Inline: false Dec 21 12:53:32.080975975 (381h24m31.95205901s) [upload=334718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:53:32.080980384 (381h24m31.952063419s) [upload=334718]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:53:32.328092505 (381h24m32.19917554s) [seg_pos={0 0}, upload=334718]: Begin upload segment... Dec 21 12:53:32.328182436 (381h24m32.199265471s) [seg_pos={0 0}, upload=334718]: Done begin upload segment. Dec 21 12:53:32.328220579 (381h24m32.199303624s) [noise=true , address=145.79.227.48:4302 , piece_id=GIFTHYKLR2K3DJLHB4VZ6T7NMYSA5TRAKPTEEJ53UVMQTYROMECA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.331668685 (381h24m32.20275173s) [noise=true , address=145.79.227.52:4445 , piece_id=SJDQEFSKBN6ERSH2WJUKMY4TEKA2XJQHDU7YBGTZWB7OQ52WE3KA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.332500137 (381h24m32.203583172s) [noise=true , address=145.79.227.47:4267 , piece_id=YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.333277057 (381h24m32.204360092s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.334224581 (381h24m32.205307617s) [noise=true , address=145.79.227.42:4121 , piece_id=PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.334844001 (381h24m32.205927046s) [noise=true , address=45.91.116.6:4974 , piece_id=ZX3KN4IETHLW3ABY6J63I35OMTQC2ZMFKO3GBPU4PY4HDS6OA22A, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.335828375 (381h24m32.20691142s) [noise=true , address=145.79.227.29:4943 , piece_id=DZZBLVTCTKGAHIFCHYNS5JLXQD2IBSP6XELBRGWS4QUBPYFMH7SA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.336610345 (381h24m32.20769338s) [noise=true , address=145.79.227.49:4338 , piece_id=I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.337469029 (381h24m32.208552074s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.338354977 (381h24m32.209438022s) [noise=true , address=145.79.227.34:4907 , piece_id=7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.339032116 (381h24m32.210115161s) [noise=true , address=145.79.227.15:4875 , piece_id=J53SUZ5OSH7B6THTLOG6762X2VDONOODZQSPT4I3BKGOCZZV3HEA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.340261366 (381h24m32.211344401s) [noise=true , address=145.79.227.14:4839 , piece_id=RMPSFFROF52HBMWI7HYYOZRYPN3YM6CCUSZJQABGOCKQOEOMGPZA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.340406363 (381h24m32.211489398s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.341633339 (381h24m32.212716374s) [noise=true , address=145.79.227.33:4870 , piece_id=GO6YIYMOGWGKS3KHIKAUBTOYTJLENKH6M7HJWLHC3OTNMUA4WJXA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.342100158 (381h24m32.213183193s) [noise=true , address=145.79.227.41:4085 , piece_id=36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.343690196 (381h24m32.214773241s) [noise=true , address=145.79.227.32:4834 , piece_id=T6AGJLKTKW4MJTMXEBAW4YBF5BSWQRKHWYKLEU4Y5U37NAMMAJTQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:34.215235315 (381h24m34.08631835s) [upload=334718]: Next returned nil segment Dec 21 12:53:34.215255653 (381h24m34.086338689s) [upload=334718]: Waiting for error group managing segments... Dec 21 12:53:34.523622594 (381h24m34.394705629s) [noise=true , address=145.79.227.49:4338 , piece_id=I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.525892566 (381h24m34.396975601s) [noise=true , address=145.79.227.14:4839 , piece_id=RMPSFFROF52HBMWI7HYYOZRYPN3YM6CCUSZJQABGOCKQOEOMGPZA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.535347309 (381h24m34.406430344s) [noise=true , address=145.79.227.41:4085 , piece_id=36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.592006026 (381h24m34.463089061s) [noise=true , address=145.79.227.34:4907 , piece_id=7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.609875664 (381h24m34.480958709s) [noise=true , address=145.79.227.32:4834 , piece_id=T6AGJLKTKW4MJTMXEBAW4YBF5BSWQRKHWYKLEU4Y5U37NAMMAJTQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.622134556 (381h24m34.493217591s) [noise=true , address=145.79.227.15:4875 , piece_id=J53SUZ5OSH7B6THTLOG6762X2VDONOODZQSPT4I3BKGOCZZV3HEA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.755361442 (381h24m34.626444487s) [noise=true , address=145.79.227.29:4943 , piece_id=DZZBLVTCTKGAHIFCHYNS5JLXQD2IBSP6XELBRGWS4QUBPYFMH7SA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.876619305 (381h24m34.74770235s) [noise=true , address=145.79.227.33:4870 , piece_id=GO6YIYMOGWGKS3KHIKAUBTOYTJLENKH6M7HJWLHC3OTNMUA4WJXA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.899187028 (381h24m34.770270073s) [noise=true , address=145.79.227.52:4445 , piece_id=SJDQEFSKBN6ERSH2WJUKMY4TEKA2XJQHDU7YBGTZWB7OQ52WE3KA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.205676652 (381h24m35.076759698s) [noise=true , address=145.79.227.42:4121 , piece_id=PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.259291003 (381h24m35.130374048s) [noise=true , address=145.79.227.48:4302 , piece_id=GIFTHYKLR2K3DJLHB4VZ6T7NMYSA5TRAKPTEEJ53UVMQTYROMECA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.326071153 (381h24m35.197154188s) [noise=true , address=45.91.116.6:4974 , piece_id=ZX3KN4IETHLW3ABY6J63I35OMTQC2ZMFKO3GBPU4PY4HDS6OA22A, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.502232413 (381h24m35.373315448s) [noise=true , address=145.79.227.47:4267 , piece_id=YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.502253854 (381h24m35.373336919s) [seg_pos={0 0}, upload=334718]: Segment reached optimal threshold of 13 pieces. Dec 21 12:53:35.502568303 (381h24m35.373651338s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 21 12:53:35.502581327 (381h24m35.373664372s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502608810 (381h24m35.373691845s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection Dec 21 12:53:35.502619169 (381h24m35.373702214s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502846553 (381h24m35.373929588s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection Dec 21 12:53:35.502853666 (381h24m35.373936701s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502873894 (381h24m35.373956929s) [seg_pos={0 0}, upload=334718]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:53:35.502965338 (381h24m35.374048383s) [upload=334718]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:53:35.502970939 (381h24m35.374053984s) [upload=334718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:53:35.794077451 (381h24m35.665160486s) [upload=334718]: Done uploading segments... Dec 21 12:53:35.794103912 (381h24m35.665186957s) [upload=334718]: Upload finished. err: <nil> Duration: 6.028701696s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X..X.........X.X.X.....X..X.....X..X....X........X..........] duration: 5.813977047s trace: https://jaeger.dev.storj.io/trace/5254881e4ca2c0fe FAIL (2025-12-21 13:01:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, Piece ID: YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A) context deadline exceeded; piecestore: (Node ID: 1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, Piece ID: I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ) context deadline exceeded; piecestore: (Node ID: 12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, Piece ID: 7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ) context deadline exceeded; piecestore: (Node ID: 17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, Piece ID: 36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q) context deadline exceeded ======LOGS====== ents... Dec 21 12:53:32.080943203 (381h24m31.952026248s) [upload=334718]: Got next segment. Inline: false Dec 21 12:53:32.080975975 (381h24m31.95205901s) [upload=334718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:53:32.080980384 (381h24m31.952063419s) [upload=334718]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:53:32.328092505 (381h24m32.19917554s) [seg_pos={0 0}, upload=334718]: Begin upload segment... Dec 21 12:53:32.328182436 (381h24m32.199265471s) [seg_pos={0 0}, upload=334718]: Done begin upload segment. Dec 21 12:53:32.328220579 (381h24m32.199303624s) [noise=true , address=145.79.227.48:4302 , piece_id=GIFTHYKLR2K3DJLHB4VZ6T7NMYSA5TRAKPTEEJ53UVMQTYROMECA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.331668685 (381h24m32.20275173s) [noise=true , address=145.79.227.52:4445 , piece_id=SJDQEFSKBN6ERSH2WJUKMY4TEKA2XJQHDU7YBGTZWB7OQ52WE3KA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.332500137 (381h24m32.203583172s) [noise=true , address=145.79.227.47:4267 , piece_id=YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.333277057 (381h24m32.204360092s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.334224581 (381h24m32.205307617s) [noise=true , address=145.79.227.42:4121 , piece_id=PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.334844001 (381h24m32.205927046s) [noise=true , address=45.91.116.6:4974 , piece_id=ZX3KN4IETHLW3ABY6J63I35OMTQC2ZMFKO3GBPU4PY4HDS6OA22A, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.335828375 (381h24m32.20691142s) [noise=true , address=145.79.227.29:4943 , piece_id=DZZBLVTCTKGAHIFCHYNS5JLXQD2IBSP6XELBRGWS4QUBPYFMH7SA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.336610345 (381h24m32.20769338s) [noise=true , address=145.79.227.49:4338 , piece_id=I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.337469029 (381h24m32.208552074s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.338354977 (381h24m32.209438022s) [noise=true , address=145.79.227.34:4907 , piece_id=7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.339032116 (381h24m32.210115161s) [noise=true , address=145.79.227.15:4875 , piece_id=J53SUZ5OSH7B6THTLOG6762X2VDONOODZQSPT4I3BKGOCZZV3HEA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.340261366 (381h24m32.211344401s) [noise=true , address=145.79.227.14:4839 , piece_id=RMPSFFROF52HBMWI7HYYOZRYPN3YM6CCUSZJQABGOCKQOEOMGPZA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.340406363 (381h24m32.211489398s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.341633339 (381h24m32.212716374s) [noise=true , address=145.79.227.33:4870 , piece_id=GO6YIYMOGWGKS3KHIKAUBTOYTJLENKH6M7HJWLHC3OTNMUA4WJXA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.342100158 (381h24m32.213183193s) [noise=true , address=145.79.227.41:4085 , piece_id=36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.343690196 (381h24m32.214773241s) [noise=true , address=145.79.227.32:4834 , piece_id=T6AGJLKTKW4MJTMXEBAW4YBF5BSWQRKHWYKLEU4Y5U37NAMMAJTQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:34.215235315 (381h24m34.08631835s) [upload=334718]: Next returned nil segment Dec 21 12:53:34.215255653 (381h24m34.086338689s) [upload=334718]: Waiting for error group managing segments... Dec 21 12:53:34.523622594 (381h24m34.394705629s) [noise=true , address=145.79.227.49:4338 , piece_id=I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.525892566 (381h24m34.396975601s) [noise=true , address=145.79.227.14:4839 , piece_id=RMPSFFROF52HBMWI7HYYOZRYPN3YM6CCUSZJQABGOCKQOEOMGPZA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.535347309 (381h24m34.406430344s) [noise=true , address=145.79.227.41:4085 , piece_id=36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.592006026 (381h24m34.463089061s) [noise=true , address=145.79.227.34:4907 , piece_id=7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.609875664 (381h24m34.480958709s) [noise=true , address=145.79.227.32:4834 , piece_id=T6AGJLKTKW4MJTMXEBAW4YBF5BSWQRKHWYKLEU4Y5U37NAMMAJTQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.622134556 (381h24m34.493217591s) [noise=true , address=145.79.227.15:4875 , piece_id=J53SUZ5OSH7B6THTLOG6762X2VDONOODZQSPT4I3BKGOCZZV3HEA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.755361442 (381h24m34.626444487s) [noise=true , address=145.79.227.29:4943 , piece_id=DZZBLVTCTKGAHIFCHYNS5JLXQD2IBSP6XELBRGWS4QUBPYFMH7SA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.876619305 (381h24m34.74770235s) [noise=true , address=145.79.227.33:4870 , piece_id=GO6YIYMOGWGKS3KHIKAUBTOYTJLENKH6M7HJWLHC3OTNMUA4WJXA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.899187028 (381h24m34.770270073s) [noise=true , address=145.79.227.52:4445 , piece_id=SJDQEFSKBN6ERSH2WJUKMY4TEKA2XJQHDU7YBGTZWB7OQ52WE3KA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.205676652 (381h24m35.076759698s) [noise=true , address=145.79.227.42:4121 , piece_id=PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.259291003 (381h24m35.130374048s) [noise=true , address=145.79.227.48:4302 , piece_id=GIFTHYKLR2K3DJLHB4VZ6T7NMYSA5TRAKPTEEJ53UVMQTYROMECA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.326071153 (381h24m35.197154188s) [noise=true , address=45.91.116.6:4974 , piece_id=ZX3KN4IETHLW3ABY6J63I35OMTQC2ZMFKO3GBPU4PY4HDS6OA22A, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.502232413 (381h24m35.373315448s) [noise=true , address=145.79.227.47:4267 , piece_id=YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.502253854 (381h24m35.373336919s) [seg_pos={0 0}, upload=334718]: Segment reached optimal threshold of 13 pieces. Dec 21 12:53:35.502568303 (381h24m35.373651338s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 21 12:53:35.502581327 (381h24m35.373664372s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502608810 (381h24m35.373691845s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection Dec 21 12:53:35.502619169 (381h24m35.373702214s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502846553 (381h24m35.373929588s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection Dec 21 12:53:35.502853666 (381h24m35.373936701s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502873894 (381h24m35.373956929s) [seg_pos={0 0}, upload=334718]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:53:35.502965338 (381h24m35.374048383s) [upload=334718]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:53:35.502970939 (381h24m35.374053984s) [upload=334718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:53:35.794077451 (381h24m35.665160486s) [upload=334718]: Done uploading segments... Dec 21 12:53:35.794103912 (381h24m35.665186957s) [upload=334718]: Upload finished. err: <nil> Duration: 5.81411984s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X..X.........X.X.X.....X..X.....X..X....X........X.........] duration: 5.818600733s trace: https://jaeger.dev.storj.io/trace/5254881e4ca2c0fe FAIL (2025-12-21 13:00:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, Piece ID: YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A) context deadline exceeded; piecestore: (Node ID: 1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, Piece ID: I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ) context deadline exceeded; piecestore: (Node ID: 12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, Piece ID: 7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ) context deadline exceeded; piecestore: (Node ID: 17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, Piece ID: 36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q) context deadline exceeded ======LOGS====== ents... Dec 21 12:53:32.080943203 (381h24m31.952026248s) [upload=334718]: Got next segment. Inline: false Dec 21 12:53:32.080975975 (381h24m31.95205901s) [upload=334718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:53:32.080980384 (381h24m31.952063419s) [upload=334718]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:53:32.328092505 (381h24m32.19917554s) [seg_pos={0 0}, upload=334718]: Begin upload segment... Dec 21 12:53:32.328182436 (381h24m32.199265471s) [seg_pos={0 0}, upload=334718]: Done begin upload segment. Dec 21 12:53:32.328220579 (381h24m32.199303624s) [noise=true , address=145.79.227.48:4302 , piece_id=GIFTHYKLR2K3DJLHB4VZ6T7NMYSA5TRAKPTEEJ53UVMQTYROMECA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.331668685 (381h24m32.20275173s) [noise=true , address=145.79.227.52:4445 , piece_id=SJDQEFSKBN6ERSH2WJUKMY4TEKA2XJQHDU7YBGTZWB7OQ52WE3KA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.332500137 (381h24m32.203583172s) [noise=true , address=145.79.227.47:4267 , piece_id=YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.333277057 (381h24m32.204360092s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.334224581 (381h24m32.205307617s) [noise=true , address=145.79.227.42:4121 , piece_id=PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.334844001 (381h24m32.205927046s) [noise=true , address=45.91.116.6:4974 , piece_id=ZX3KN4IETHLW3ABY6J63I35OMTQC2ZMFKO3GBPU4PY4HDS6OA22A, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.335828375 (381h24m32.20691142s) [noise=true , address=145.79.227.29:4943 , piece_id=DZZBLVTCTKGAHIFCHYNS5JLXQD2IBSP6XELBRGWS4QUBPYFMH7SA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.336610345 (381h24m32.20769338s) [noise=true , address=145.79.227.49:4338 , piece_id=I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.337469029 (381h24m32.208552074s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.338354977 (381h24m32.209438022s) [noise=true , address=145.79.227.34:4907 , piece_id=7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.339032116 (381h24m32.210115161s) [noise=true , address=145.79.227.15:4875 , piece_id=J53SUZ5OSH7B6THTLOG6762X2VDONOODZQSPT4I3BKGOCZZV3HEA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.340261366 (381h24m32.211344401s) [noise=true , address=145.79.227.14:4839 , piece_id=RMPSFFROF52HBMWI7HYYOZRYPN3YM6CCUSZJQABGOCKQOEOMGPZA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.340406363 (381h24m32.211489398s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.341633339 (381h24m32.212716374s) [noise=true , address=145.79.227.33:4870 , piece_id=GO6YIYMOGWGKS3KHIKAUBTOYTJLENKH6M7HJWLHC3OTNMUA4WJXA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.342100158 (381h24m32.213183193s) [noise=true , address=145.79.227.41:4085 , piece_id=36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.343690196 (381h24m32.214773241s) [noise=true , address=145.79.227.32:4834 , piece_id=T6AGJLKTKW4MJTMXEBAW4YBF5BSWQRKHWYKLEU4Y5U37NAMMAJTQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:34.215235315 (381h24m34.08631835s) [upload=334718]: Next returned nil segment Dec 21 12:53:34.215255653 (381h24m34.086338689s) [upload=334718]: Waiting for error group managing segments... Dec 21 12:53:34.523622594 (381h24m34.394705629s) [noise=true , address=145.79.227.49:4338 , piece_id=I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.525892566 (381h24m34.396975601s) [noise=true , address=145.79.227.14:4839 , piece_id=RMPSFFROF52HBMWI7HYYOZRYPN3YM6CCUSZJQABGOCKQOEOMGPZA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.535347309 (381h24m34.406430344s) [noise=true , address=145.79.227.41:4085 , piece_id=36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.592006026 (381h24m34.463089061s) [noise=true , address=145.79.227.34:4907 , piece_id=7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.609875664 (381h24m34.480958709s) [noise=true , address=145.79.227.32:4834 , piece_id=T6AGJLKTKW4MJTMXEBAW4YBF5BSWQRKHWYKLEU4Y5U37NAMMAJTQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.622134556 (381h24m34.493217591s) [noise=true , address=145.79.227.15:4875 , piece_id=J53SUZ5OSH7B6THTLOG6762X2VDONOODZQSPT4I3BKGOCZZV3HEA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.755361442 (381h24m34.626444487s) [noise=true , address=145.79.227.29:4943 , piece_id=DZZBLVTCTKGAHIFCHYNS5JLXQD2IBSP6XELBRGWS4QUBPYFMH7SA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.876619305 (381h24m34.74770235s) [noise=true , address=145.79.227.33:4870 , piece_id=GO6YIYMOGWGKS3KHIKAUBTOYTJLENKH6M7HJWLHC3OTNMUA4WJXA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.899187028 (381h24m34.770270073s) [noise=true , address=145.79.227.52:4445 , piece_id=SJDQEFSKBN6ERSH2WJUKMY4TEKA2XJQHDU7YBGTZWB7OQ52WE3KA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.205676652 (381h24m35.076759698s) [noise=true , address=145.79.227.42:4121 , piece_id=PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.259291003 (381h24m35.130374048s) [noise=true , address=145.79.227.48:4302 , piece_id=GIFTHYKLR2K3DJLHB4VZ6T7NMYSA5TRAKPTEEJ53UVMQTYROMECA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.326071153 (381h24m35.197154188s) [noise=true , address=45.91.116.6:4974 , piece_id=ZX3KN4IETHLW3ABY6J63I35OMTQC2ZMFKO3GBPU4PY4HDS6OA22A, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.502232413 (381h24m35.373315448s) [noise=true , address=145.79.227.47:4267 , piece_id=YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.502253854 (381h24m35.373336919s) [seg_pos={0 0}, upload=334718]: Segment reached optimal threshold of 13 pieces. Dec 21 12:53:35.502568303 (381h24m35.373651338s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 21 12:53:35.502581327 (381h24m35.373664372s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502608810 (381h24m35.373691845s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection Dec 21 12:53:35.502619169 (381h24m35.373702214s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502846553 (381h24m35.373929588s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection Dec 21 12:53:35.502853666 (381h24m35.373936701s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502873894 (381h24m35.373956929s) [seg_pos={0 0}, upload=334718]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:53:35.502965338 (381h24m35.374048383s) [upload=334718]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:53:35.502970939 (381h24m35.374053984s) [upload=334718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:53:35.794077451 (381h24m35.665160486s) [upload=334718]: Done uploading segments... Dec 21 12:53:35.794103912 (381h24m35.665186957s) [upload=334718]: Upload finished. err: <nil> Duration: 5.818648373s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X.X..X.........X.X.X.....X..X.....X..X....X........X........] duration: 5.809383018s trace: https://jaeger.dev.storj.io/trace/5254881e4ca2c0fe FAIL (2025-12-21 12:59:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, Piece ID: YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A) context deadline exceeded; piecestore: (Node ID: 1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, Piece ID: I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ) context deadline exceeded; piecestore: (Node ID: 12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, Piece ID: 7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ) context deadline exceeded; piecestore: (Node ID: 17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, Piece ID: 36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q) context deadline exceeded ======LOGS====== ents... Dec 21 12:53:32.080943203 (381h24m31.952026248s) [upload=334718]: Got next segment. Inline: false Dec 21 12:53:32.080975975 (381h24m31.95205901s) [upload=334718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:53:32.080980384 (381h24m31.952063419s) [upload=334718]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:53:32.328092505 (381h24m32.19917554s) [seg_pos={0 0}, upload=334718]: Begin upload segment... Dec 21 12:53:32.328182436 (381h24m32.199265471s) [seg_pos={0 0}, upload=334718]: Done begin upload segment. Dec 21 12:53:32.328220579 (381h24m32.199303624s) [noise=true , address=145.79.227.48:4302 , piece_id=GIFTHYKLR2K3DJLHB4VZ6T7NMYSA5TRAKPTEEJ53UVMQTYROMECA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.331668685 (381h24m32.20275173s) [noise=true , address=145.79.227.52:4445 , piece_id=SJDQEFSKBN6ERSH2WJUKMY4TEKA2XJQHDU7YBGTZWB7OQ52WE3KA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.332500137 (381h24m32.203583172s) [noise=true , address=145.79.227.47:4267 , piece_id=YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.333277057 (381h24m32.204360092s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.334224581 (381h24m32.205307617s) [noise=true , address=145.79.227.42:4121 , piece_id=PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.334844001 (381h24m32.205927046s) [noise=true , address=45.91.116.6:4974 , piece_id=ZX3KN4IETHLW3ABY6J63I35OMTQC2ZMFKO3GBPU4PY4HDS6OA22A, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.335828375 (381h24m32.20691142s) [noise=true , address=145.79.227.29:4943 , piece_id=DZZBLVTCTKGAHIFCHYNS5JLXQD2IBSP6XELBRGWS4QUBPYFMH7SA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.336610345 (381h24m32.20769338s) [noise=true , address=145.79.227.49:4338 , piece_id=I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.337469029 (381h24m32.208552074s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.338354977 (381h24m32.209438022s) [noise=true , address=145.79.227.34:4907 , piece_id=7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.339032116 (381h24m32.210115161s) [noise=true , address=145.79.227.15:4875 , piece_id=J53SUZ5OSH7B6THTLOG6762X2VDONOODZQSPT4I3BKGOCZZV3HEA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.340261366 (381h24m32.211344401s) [noise=true , address=145.79.227.14:4839 , piece_id=RMPSFFROF52HBMWI7HYYOZRYPN3YM6CCUSZJQABGOCKQOEOMGPZA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.340406363 (381h24m32.211489398s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.341633339 (381h24m32.212716374s) [noise=true , address=145.79.227.33:4870 , piece_id=GO6YIYMOGWGKS3KHIKAUBTOYTJLENKH6M7HJWLHC3OTNMUA4WJXA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.342100158 (381h24m32.213183193s) [noise=true , address=145.79.227.41:4085 , piece_id=36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.343690196 (381h24m32.214773241s) [noise=true , address=145.79.227.32:4834 , piece_id=T6AGJLKTKW4MJTMXEBAW4YBF5BSWQRKHWYKLEU4Y5U37NAMMAJTQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:34.215235315 (381h24m34.08631835s) [upload=334718]: Next returned nil segment Dec 21 12:53:34.215255653 (381h24m34.086338689s) [upload=334718]: Waiting for error group managing segments... Dec 21 12:53:34.523622594 (381h24m34.394705629s) [noise=true , address=145.79.227.49:4338 , piece_id=I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.525892566 (381h24m34.396975601s) [noise=true , address=145.79.227.14:4839 , piece_id=RMPSFFROF52HBMWI7HYYOZRYPN3YM6CCUSZJQABGOCKQOEOMGPZA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.535347309 (381h24m34.406430344s) [noise=true , address=145.79.227.41:4085 , piece_id=36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.592006026 (381h24m34.463089061s) [noise=true , address=145.79.227.34:4907 , piece_id=7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.609875664 (381h24m34.480958709s) [noise=true , address=145.79.227.32:4834 , piece_id=T6AGJLKTKW4MJTMXEBAW4YBF5BSWQRKHWYKLEU4Y5U37NAMMAJTQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.622134556 (381h24m34.493217591s) [noise=true , address=145.79.227.15:4875 , piece_id=J53SUZ5OSH7B6THTLOG6762X2VDONOODZQSPT4I3BKGOCZZV3HEA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.755361442 (381h24m34.626444487s) [noise=true , address=145.79.227.29:4943 , piece_id=DZZBLVTCTKGAHIFCHYNS5JLXQD2IBSP6XELBRGWS4QUBPYFMH7SA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.876619305 (381h24m34.74770235s) [noise=true , address=145.79.227.33:4870 , piece_id=GO6YIYMOGWGKS3KHIKAUBTOYTJLENKH6M7HJWLHC3OTNMUA4WJXA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.899187028 (381h24m34.770270073s) [noise=true , address=145.79.227.52:4445 , piece_id=SJDQEFSKBN6ERSH2WJUKMY4TEKA2XJQHDU7YBGTZWB7OQ52WE3KA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.205676652 (381h24m35.076759698s) [noise=true , address=145.79.227.42:4121 , piece_id=PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.259291003 (381h24m35.130374048s) [noise=true , address=145.79.227.48:4302 , piece_id=GIFTHYKLR2K3DJLHB4VZ6T7NMYSA5TRAKPTEEJ53UVMQTYROMECA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.326071153 (381h24m35.197154188s) [noise=true , address=45.91.116.6:4974 , piece_id=ZX3KN4IETHLW3ABY6J63I35OMTQC2ZMFKO3GBPU4PY4HDS6OA22A, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.502232413 (381h24m35.373315448s) [noise=true , address=145.79.227.47:4267 , piece_id=YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.502253854 (381h24m35.373336919s) [seg_pos={0 0}, upload=334718]: Segment reached optimal threshold of 13 pieces. Dec 21 12:53:35.502568303 (381h24m35.373651338s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 21 12:53:35.502581327 (381h24m35.373664372s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502608810 (381h24m35.373691845s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection Dec 21 12:53:35.502619169 (381h24m35.373702214s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502846553 (381h24m35.373929588s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection Dec 21 12:53:35.502853666 (381h24m35.373936701s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502873894 (381h24m35.373956929s) [seg_pos={0 0}, upload=334718]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:53:35.502965338 (381h24m35.374048383s) [upload=334718]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:53:35.502970939 (381h24m35.374053984s) [upload=334718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:53:35.794077451 (381h24m35.665160486s) [upload=334718]: Done uploading segments... Dec 21 12:53:35.794103912 (381h24m35.665186957s) [upload=334718]: Upload finished. err: <nil> Duration: 5.809407927s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.X.X..X.........X.X.X.....X..X.....X..X....X........X.......] duration: 5.902783076s trace: https://jaeger.dev.storj.io/trace/5254881e4ca2c0fe FAIL (2025-12-21 12:58:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, Piece ID: YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A) context deadline exceeded; piecestore: (Node ID: 1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, Piece ID: I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ) context deadline exceeded; piecestore: (Node ID: 12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, Piece ID: 7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ) context deadline exceeded; piecestore: (Node ID: 17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, Piece ID: 36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q) context deadline exceeded ======LOGS====== ents... Dec 21 12:53:32.080943203 (381h24m31.952026248s) [upload=334718]: Got next segment. Inline: false Dec 21 12:53:32.080975975 (381h24m31.95205901s) [upload=334718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:53:32.080980384 (381h24m31.952063419s) [upload=334718]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:53:32.328092505 (381h24m32.19917554s) [seg_pos={0 0}, upload=334718]: Begin upload segment... Dec 21 12:53:32.328182436 (381h24m32.199265471s) [seg_pos={0 0}, upload=334718]: Done begin upload segment. Dec 21 12:53:32.328220579 (381h24m32.199303624s) [noise=true , address=145.79.227.48:4302 , piece_id=GIFTHYKLR2K3DJLHB4VZ6T7NMYSA5TRAKPTEEJ53UVMQTYROMECA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.331668685 (381h24m32.20275173s) [noise=true , address=145.79.227.52:4445 , piece_id=SJDQEFSKBN6ERSH2WJUKMY4TEKA2XJQHDU7YBGTZWB7OQ52WE3KA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.332500137 (381h24m32.203583172s) [noise=true , address=145.79.227.47:4267 , piece_id=YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.333277057 (381h24m32.204360092s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.334224581 (381h24m32.205307617s) [noise=true , address=145.79.227.42:4121 , piece_id=PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.334844001 (381h24m32.205927046s) [noise=true , address=45.91.116.6:4974 , piece_id=ZX3KN4IETHLW3ABY6J63I35OMTQC2ZMFKO3GBPU4PY4HDS6OA22A, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.335828375 (381h24m32.20691142s) [noise=true , address=145.79.227.29:4943 , piece_id=DZZBLVTCTKGAHIFCHYNS5JLXQD2IBSP6XELBRGWS4QUBPYFMH7SA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.336610345 (381h24m32.20769338s) [noise=true , address=145.79.227.49:4338 , piece_id=I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.337469029 (381h24m32.208552074s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.338354977 (381h24m32.209438022s) [noise=true , address=145.79.227.34:4907 , piece_id=7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.339032116 (381h24m32.210115161s) [noise=true , address=145.79.227.15:4875 , piece_id=J53SUZ5OSH7B6THTLOG6762X2VDONOODZQSPT4I3BKGOCZZV3HEA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.340261366 (381h24m32.211344401s) [noise=true , address=145.79.227.14:4839 , piece_id=RMPSFFROF52HBMWI7HYYOZRYPN3YM6CCUSZJQABGOCKQOEOMGPZA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.340406363 (381h24m32.211489398s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.341633339 (381h24m32.212716374s) [noise=true , address=145.79.227.33:4870 , piece_id=GO6YIYMOGWGKS3KHIKAUBTOYTJLENKH6M7HJWLHC3OTNMUA4WJXA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.342100158 (381h24m32.213183193s) [noise=true , address=145.79.227.41:4085 , piece_id=36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.343690196 (381h24m32.214773241s) [noise=true , address=145.79.227.32:4834 , piece_id=T6AGJLKTKW4MJTMXEBAW4YBF5BSWQRKHWYKLEU4Y5U37NAMMAJTQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:34.215235315 (381h24m34.08631835s) [upload=334718]: Next returned nil segment Dec 21 12:53:34.215255653 (381h24m34.086338689s) [upload=334718]: Waiting for error group managing segments... Dec 21 12:53:34.523622594 (381h24m34.394705629s) [noise=true , address=145.79.227.49:4338 , piece_id=I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.525892566 (381h24m34.396975601s) [noise=true , address=145.79.227.14:4839 , piece_id=RMPSFFROF52HBMWI7HYYOZRYPN3YM6CCUSZJQABGOCKQOEOMGPZA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.535347309 (381h24m34.406430344s) [noise=true , address=145.79.227.41:4085 , piece_id=36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.592006026 (381h24m34.463089061s) [noise=true , address=145.79.227.34:4907 , piece_id=7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.609875664 (381h24m34.480958709s) [noise=true , address=145.79.227.32:4834 , piece_id=T6AGJLKTKW4MJTMXEBAW4YBF5BSWQRKHWYKLEU4Y5U37NAMMAJTQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.622134556 (381h24m34.493217591s) [noise=true , address=145.79.227.15:4875 , piece_id=J53SUZ5OSH7B6THTLOG6762X2VDONOODZQSPT4I3BKGOCZZV3HEA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.755361442 (381h24m34.626444487s) [noise=true , address=145.79.227.29:4943 , piece_id=DZZBLVTCTKGAHIFCHYNS5JLXQD2IBSP6XELBRGWS4QUBPYFMH7SA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.876619305 (381h24m34.74770235s) [noise=true , address=145.79.227.33:4870 , piece_id=GO6YIYMOGWGKS3KHIKAUBTOYTJLENKH6M7HJWLHC3OTNMUA4WJXA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.899187028 (381h24m34.770270073s) [noise=true , address=145.79.227.52:4445 , piece_id=SJDQEFSKBN6ERSH2WJUKMY4TEKA2XJQHDU7YBGTZWB7OQ52WE3KA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.205676652 (381h24m35.076759698s) [noise=true , address=145.79.227.42:4121 , piece_id=PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.259291003 (381h24m35.130374048s) [noise=true , address=145.79.227.48:4302 , piece_id=GIFTHYKLR2K3DJLHB4VZ6T7NMYSA5TRAKPTEEJ53UVMQTYROMECA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.326071153 (381h24m35.197154188s) [noise=true , address=45.91.116.6:4974 , piece_id=ZX3KN4IETHLW3ABY6J63I35OMTQC2ZMFKO3GBPU4PY4HDS6OA22A, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.502232413 (381h24m35.373315448s) [noise=true , address=145.79.227.47:4267 , piece_id=YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.502253854 (381h24m35.373336919s) [seg_pos={0 0}, upload=334718]: Segment reached optimal threshold of 13 pieces. Dec 21 12:53:35.502568303 (381h24m35.373651338s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 21 12:53:35.502581327 (381h24m35.373664372s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502608810 (381h24m35.373691845s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection Dec 21 12:53:35.502619169 (381h24m35.373702214s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502846553 (381h24m35.373929588s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection Dec 21 12:53:35.502853666 (381h24m35.373936701s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502873894 (381h24m35.373956929s) [seg_pos={0 0}, upload=334718]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:53:35.502965338 (381h24m35.374048383s) [upload=334718]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:53:35.502970939 (381h24m35.374053984s) [upload=334718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:53:35.794077451 (381h24m35.665160486s) [upload=334718]: Done uploading segments... Dec 21 12:53:35.794103912 (381h24m35.665186957s) [upload=334718]: Upload finished. err: <nil> Duration: 5.902813774s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X.X.X..X.........X.X.X.....X..X.....X..X....X........X......] duration: 5.959185594s trace: https://jaeger.dev.storj.io/trace/5254881e4ca2c0fe FAIL (2025-12-21 12:57:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, Piece ID: YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A) context deadline exceeded; piecestore: (Node ID: 1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, Piece ID: I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ) context deadline exceeded; piecestore: (Node ID: 12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, Piece ID: 7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ) context deadline exceeded; piecestore: (Node ID: 17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, Piece ID: 36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q) context deadline exceeded ======LOGS====== ents... Dec 21 12:53:32.080943203 (381h24m31.952026248s) [upload=334718]: Got next segment. Inline: false Dec 21 12:53:32.080975975 (381h24m31.95205901s) [upload=334718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:53:32.080980384 (381h24m31.952063419s) [upload=334718]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:53:32.328092505 (381h24m32.19917554s) [seg_pos={0 0}, upload=334718]: Begin upload segment... Dec 21 12:53:32.328182436 (381h24m32.199265471s) [seg_pos={0 0}, upload=334718]: Done begin upload segment. Dec 21 12:53:32.328220579 (381h24m32.199303624s) [noise=true , address=145.79.227.48:4302 , piece_id=GIFTHYKLR2K3DJLHB4VZ6T7NMYSA5TRAKPTEEJ53UVMQTYROMECA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.331668685 (381h24m32.20275173s) [noise=true , address=145.79.227.52:4445 , piece_id=SJDQEFSKBN6ERSH2WJUKMY4TEKA2XJQHDU7YBGTZWB7OQ52WE3KA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.332500137 (381h24m32.203583172s) [noise=true , address=145.79.227.47:4267 , piece_id=YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.333277057 (381h24m32.204360092s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.334224581 (381h24m32.205307617s) [noise=true , address=145.79.227.42:4121 , piece_id=PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.334844001 (381h24m32.205927046s) [noise=true , address=45.91.116.6:4974 , piece_id=ZX3KN4IETHLW3ABY6J63I35OMTQC2ZMFKO3GBPU4PY4HDS6OA22A, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.335828375 (381h24m32.20691142s) [noise=true , address=145.79.227.29:4943 , piece_id=DZZBLVTCTKGAHIFCHYNS5JLXQD2IBSP6XELBRGWS4QUBPYFMH7SA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.336610345 (381h24m32.20769338s) [noise=true , address=145.79.227.49:4338 , piece_id=I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.337469029 (381h24m32.208552074s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.338354977 (381h24m32.209438022s) [noise=true , address=145.79.227.34:4907 , piece_id=7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.339032116 (381h24m32.210115161s) [noise=true , address=145.79.227.15:4875 , piece_id=J53SUZ5OSH7B6THTLOG6762X2VDONOODZQSPT4I3BKGOCZZV3HEA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.340261366 (381h24m32.211344401s) [noise=true , address=145.79.227.14:4839 , piece_id=RMPSFFROF52HBMWI7HYYOZRYPN3YM6CCUSZJQABGOCKQOEOMGPZA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.340406363 (381h24m32.211489398s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.341633339 (381h24m32.212716374s) [noise=true , address=145.79.227.33:4870 , piece_id=GO6YIYMOGWGKS3KHIKAUBTOYTJLENKH6M7HJWLHC3OTNMUA4WJXA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.342100158 (381h24m32.213183193s) [noise=true , address=145.79.227.41:4085 , piece_id=36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.343690196 (381h24m32.214773241s) [noise=true , address=145.79.227.32:4834 , piece_id=T6AGJLKTKW4MJTMXEBAW4YBF5BSWQRKHWYKLEU4Y5U37NAMMAJTQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:34.215235315 (381h24m34.08631835s) [upload=334718]: Next returned nil segment Dec 21 12:53:34.215255653 (381h24m34.086338689s) [upload=334718]: Waiting for error group managing segments... Dec 21 12:53:34.523622594 (381h24m34.394705629s) [noise=true , address=145.79.227.49:4338 , piece_id=I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.525892566 (381h24m34.396975601s) [noise=true , address=145.79.227.14:4839 , piece_id=RMPSFFROF52HBMWI7HYYOZRYPN3YM6CCUSZJQABGOCKQOEOMGPZA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.535347309 (381h24m34.406430344s) [noise=true , address=145.79.227.41:4085 , piece_id=36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.592006026 (381h24m34.463089061s) [noise=true , address=145.79.227.34:4907 , piece_id=7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.609875664 (381h24m34.480958709s) [noise=true , address=145.79.227.32:4834 , piece_id=T6AGJLKTKW4MJTMXEBAW4YBF5BSWQRKHWYKLEU4Y5U37NAMMAJTQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.622134556 (381h24m34.493217591s) [noise=true , address=145.79.227.15:4875 , piece_id=J53SUZ5OSH7B6THTLOG6762X2VDONOODZQSPT4I3BKGOCZZV3HEA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.755361442 (381h24m34.626444487s) [noise=true , address=145.79.227.29:4943 , piece_id=DZZBLVTCTKGAHIFCHYNS5JLXQD2IBSP6XELBRGWS4QUBPYFMH7SA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.876619305 (381h24m34.74770235s) [noise=true , address=145.79.227.33:4870 , piece_id=GO6YIYMOGWGKS3KHIKAUBTOYTJLENKH6M7HJWLHC3OTNMUA4WJXA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.899187028 (381h24m34.770270073s) [noise=true , address=145.79.227.52:4445 , piece_id=SJDQEFSKBN6ERSH2WJUKMY4TEKA2XJQHDU7YBGTZWB7OQ52WE3KA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.205676652 (381h24m35.076759698s) [noise=true , address=145.79.227.42:4121 , piece_id=PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.259291003 (381h24m35.130374048s) [noise=true , address=145.79.227.48:4302 , piece_id=GIFTHYKLR2K3DJLHB4VZ6T7NMYSA5TRAKPTEEJ53UVMQTYROMECA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.326071153 (381h24m35.197154188s) [noise=true , address=45.91.116.6:4974 , piece_id=ZX3KN4IETHLW3ABY6J63I35OMTQC2ZMFKO3GBPU4PY4HDS6OA22A, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.502232413 (381h24m35.373315448s) [noise=true , address=145.79.227.47:4267 , piece_id=YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.502253854 (381h24m35.373336919s) [seg_pos={0 0}, upload=334718]: Segment reached optimal threshold of 13 pieces. Dec 21 12:53:35.502568303 (381h24m35.373651338s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 21 12:53:35.502581327 (381h24m35.373664372s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502608810 (381h24m35.373691845s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection Dec 21 12:53:35.502619169 (381h24m35.373702214s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502846553 (381h24m35.373929588s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection Dec 21 12:53:35.502853666 (381h24m35.373936701s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502873894 (381h24m35.373956929s) [seg_pos={0 0}, upload=334718]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:53:35.502965338 (381h24m35.374048383s) [upload=334718]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:53:35.502970939 (381h24m35.374053984s) [upload=334718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:53:35.794077451 (381h24m35.665160486s) [upload=334718]: Done uploading segments... Dec 21 12:53:35.794103912 (381h24m35.665186957s) [upload=334718]: Upload finished. err: <nil> Duration: 5.959203728s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [XX.X.X..X.........X.X.X.....X..X.....X..X....X........X.....] duration: 5.951966461s trace: https://jaeger.dev.storj.io/trace/5254881e4ca2c0fe FAIL (2025-12-21 12:57:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, Piece ID: YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A) context deadline exceeded; piecestore: (Node ID: 1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, Piece ID: I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ) context deadline exceeded; piecestore: (Node ID: 12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, Piece ID: 7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ) context deadline exceeded; piecestore: (Node ID: 17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, Piece ID: 36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q) context deadline exceeded ======LOGS====== ents... Dec 21 12:53:32.080943203 (381h24m31.952026248s) [upload=334718]: Got next segment. Inline: false Dec 21 12:53:32.080975975 (381h24m31.95205901s) [upload=334718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:53:32.080980384 (381h24m31.952063419s) [upload=334718]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:53:32.328092505 (381h24m32.19917554s) [seg_pos={0 0}, upload=334718]: Begin upload segment... Dec 21 12:53:32.328182436 (381h24m32.199265471s) [seg_pos={0 0}, upload=334718]: Done begin upload segment. Dec 21 12:53:32.328220579 (381h24m32.199303624s) [noise=true , address=145.79.227.48:4302 , piece_id=GIFTHYKLR2K3DJLHB4VZ6T7NMYSA5TRAKPTEEJ53UVMQTYROMECA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.331668685 (381h24m32.20275173s) [noise=true , address=145.79.227.52:4445 , piece_id=SJDQEFSKBN6ERSH2WJUKMY4TEKA2XJQHDU7YBGTZWB7OQ52WE3KA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.332500137 (381h24m32.203583172s) [noise=true , address=145.79.227.47:4267 , piece_id=YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.333277057 (381h24m32.204360092s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.334224581 (381h24m32.205307617s) [noise=true , address=145.79.227.42:4121 , piece_id=PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.334844001 (381h24m32.205927046s) [noise=true , address=45.91.116.6:4974 , piece_id=ZX3KN4IETHLW3ABY6J63I35OMTQC2ZMFKO3GBPU4PY4HDS6OA22A, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.335828375 (381h24m32.20691142s) [noise=true , address=145.79.227.29:4943 , piece_id=DZZBLVTCTKGAHIFCHYNS5JLXQD2IBSP6XELBRGWS4QUBPYFMH7SA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.336610345 (381h24m32.20769338s) [noise=true , address=145.79.227.49:4338 , piece_id=I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.337469029 (381h24m32.208552074s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.338354977 (381h24m32.209438022s) [noise=true , address=145.79.227.34:4907 , piece_id=7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.339032116 (381h24m32.210115161s) [noise=true , address=145.79.227.15:4875 , piece_id=J53SUZ5OSH7B6THTLOG6762X2VDONOODZQSPT4I3BKGOCZZV3HEA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.340261366 (381h24m32.211344401s) [noise=true , address=145.79.227.14:4839 , piece_id=RMPSFFROF52HBMWI7HYYOZRYPN3YM6CCUSZJQABGOCKQOEOMGPZA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.340406363 (381h24m32.211489398s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.341633339 (381h24m32.212716374s) [noise=true , address=145.79.227.33:4870 , piece_id=GO6YIYMOGWGKS3KHIKAUBTOYTJLENKH6M7HJWLHC3OTNMUA4WJXA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.342100158 (381h24m32.213183193s) [noise=true , address=145.79.227.41:4085 , piece_id=36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.343690196 (381h24m32.214773241s) [noise=true , address=145.79.227.32:4834 , piece_id=T6AGJLKTKW4MJTMXEBAW4YBF5BSWQRKHWYKLEU4Y5U37NAMMAJTQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:34.215235315 (381h24m34.08631835s) [upload=334718]: Next returned nil segment Dec 21 12:53:34.215255653 (381h24m34.086338689s) [upload=334718]: Waiting for error group managing segments... Dec 21 12:53:34.523622594 (381h24m34.394705629s) [noise=true , address=145.79.227.49:4338 , piece_id=I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.525892566 (381h24m34.396975601s) [noise=true , address=145.79.227.14:4839 , piece_id=RMPSFFROF52HBMWI7HYYOZRYPN3YM6CCUSZJQABGOCKQOEOMGPZA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.535347309 (381h24m34.406430344s) [noise=true , address=145.79.227.41:4085 , piece_id=36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.592006026 (381h24m34.463089061s) [noise=true , address=145.79.227.34:4907 , piece_id=7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.609875664 (381h24m34.480958709s) [noise=true , address=145.79.227.32:4834 , piece_id=T6AGJLKTKW4MJTMXEBAW4YBF5BSWQRKHWYKLEU4Y5U37NAMMAJTQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.622134556 (381h24m34.493217591s) [noise=true , address=145.79.227.15:4875 , piece_id=J53SUZ5OSH7B6THTLOG6762X2VDONOODZQSPT4I3BKGOCZZV3HEA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.755361442 (381h24m34.626444487s) [noise=true , address=145.79.227.29:4943 , piece_id=DZZBLVTCTKGAHIFCHYNS5JLXQD2IBSP6XELBRGWS4QUBPYFMH7SA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.876619305 (381h24m34.74770235s) [noise=true , address=145.79.227.33:4870 , piece_id=GO6YIYMOGWGKS3KHIKAUBTOYTJLENKH6M7HJWLHC3OTNMUA4WJXA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.899187028 (381h24m34.770270073s) [noise=true , address=145.79.227.52:4445 , piece_id=SJDQEFSKBN6ERSH2WJUKMY4TEKA2XJQHDU7YBGTZWB7OQ52WE3KA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.205676652 (381h24m35.076759698s) [noise=true , address=145.79.227.42:4121 , piece_id=PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.259291003 (381h24m35.130374048s) [noise=true , address=145.79.227.48:4302 , piece_id=GIFTHYKLR2K3DJLHB4VZ6T7NMYSA5TRAKPTEEJ53UVMQTYROMECA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.326071153 (381h24m35.197154188s) [noise=true , address=45.91.116.6:4974 , piece_id=ZX3KN4IETHLW3ABY6J63I35OMTQC2ZMFKO3GBPU4PY4HDS6OA22A, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.502232413 (381h24m35.373315448s) [noise=true , address=145.79.227.47:4267 , piece_id=YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.502253854 (381h24m35.373336919s) [seg_pos={0 0}, upload=334718]: Segment reached optimal threshold of 13 pieces. Dec 21 12:53:35.502568303 (381h24m35.373651338s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 21 12:53:35.502581327 (381h24m35.373664372s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502608810 (381h24m35.373691845s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection Dec 21 12:53:35.502619169 (381h24m35.373702214s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502846553 (381h24m35.373929588s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection Dec 21 12:53:35.502853666 (381h24m35.373936701s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502873894 (381h24m35.373956929s) [seg_pos={0 0}, upload=334718]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:53:35.502965338 (381h24m35.374048383s) [upload=334718]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:53:35.502970939 (381h24m35.374053984s) [upload=334718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:53:35.794077451 (381h24m35.665160486s) [upload=334718]: Done uploading segments... Dec 21 12:53:35.794103912 (381h24m35.665186957s) [upload=334718]: Upload finished. err: <nil> Duration: 5.95199199s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [XXX.X.X..X.........X.X.X.....X..X.....X..X....X........X....] duration: 5.738617259s trace: https://jaeger.dev.storj.io/trace/5254881e4ca2c0fe FAIL (2025-12-21 12:56:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, Piece ID: YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A) context deadline exceeded; piecestore: (Node ID: 1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, Piece ID: I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ) context deadline exceeded; piecestore: (Node ID: 12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, Piece ID: 7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ) context deadline exceeded; piecestore: (Node ID: 17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, Piece ID: 36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q) context deadline exceeded ======LOGS====== ents... Dec 21 12:53:32.080943203 (381h24m31.952026248s) [upload=334718]: Got next segment. Inline: false Dec 21 12:53:32.080975975 (381h24m31.95205901s) [upload=334718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:53:32.080980384 (381h24m31.952063419s) [upload=334718]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:53:32.328092505 (381h24m32.19917554s) [seg_pos={0 0}, upload=334718]: Begin upload segment... Dec 21 12:53:32.328182436 (381h24m32.199265471s) [seg_pos={0 0}, upload=334718]: Done begin upload segment. Dec 21 12:53:32.328220579 (381h24m32.199303624s) [noise=true , address=145.79.227.48:4302 , piece_id=GIFTHYKLR2K3DJLHB4VZ6T7NMYSA5TRAKPTEEJ53UVMQTYROMECA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.331668685 (381h24m32.20275173s) [noise=true , address=145.79.227.52:4445 , piece_id=SJDQEFSKBN6ERSH2WJUKMY4TEKA2XJQHDU7YBGTZWB7OQ52WE3KA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.332500137 (381h24m32.203583172s) [noise=true , address=145.79.227.47:4267 , piece_id=YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.333277057 (381h24m32.204360092s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.334224581 (381h24m32.205307617s) [noise=true , address=145.79.227.42:4121 , piece_id=PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.334844001 (381h24m32.205927046s) [noise=true , address=45.91.116.6:4974 , piece_id=ZX3KN4IETHLW3ABY6J63I35OMTQC2ZMFKO3GBPU4PY4HDS6OA22A, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.335828375 (381h24m32.20691142s) [noise=true , address=145.79.227.29:4943 , piece_id=DZZBLVTCTKGAHIFCHYNS5JLXQD2IBSP6XELBRGWS4QUBPYFMH7SA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.336610345 (381h24m32.20769338s) [noise=true , address=145.79.227.49:4338 , piece_id=I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.337469029 (381h24m32.208552074s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.338354977 (381h24m32.209438022s) [noise=true , address=145.79.227.34:4907 , piece_id=7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.339032116 (381h24m32.210115161s) [noise=true , address=145.79.227.15:4875 , piece_id=J53SUZ5OSH7B6THTLOG6762X2VDONOODZQSPT4I3BKGOCZZV3HEA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.340261366 (381h24m32.211344401s) [noise=true , address=145.79.227.14:4839 , piece_id=RMPSFFROF52HBMWI7HYYOZRYPN3YM6CCUSZJQABGOCKQOEOMGPZA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.340406363 (381h24m32.211489398s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.341633339 (381h24m32.212716374s) [noise=true , address=145.79.227.33:4870 , piece_id=GO6YIYMOGWGKS3KHIKAUBTOYTJLENKH6M7HJWLHC3OTNMUA4WJXA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.342100158 (381h24m32.213183193s) [noise=true , address=145.79.227.41:4085 , piece_id=36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.343690196 (381h24m32.214773241s) [noise=true , address=145.79.227.32:4834 , piece_id=T6AGJLKTKW4MJTMXEBAW4YBF5BSWQRKHWYKLEU4Y5U37NAMMAJTQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:34.215235315 (381h24m34.08631835s) [upload=334718]: Next returned nil segment Dec 21 12:53:34.215255653 (381h24m34.086338689s) [upload=334718]: Waiting for error group managing segments... Dec 21 12:53:34.523622594 (381h24m34.394705629s) [noise=true , address=145.79.227.49:4338 , piece_id=I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.525892566 (381h24m34.396975601s) [noise=true , address=145.79.227.14:4839 , piece_id=RMPSFFROF52HBMWI7HYYOZRYPN3YM6CCUSZJQABGOCKQOEOMGPZA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.535347309 (381h24m34.406430344s) [noise=true , address=145.79.227.41:4085 , piece_id=36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.592006026 (381h24m34.463089061s) [noise=true , address=145.79.227.34:4907 , piece_id=7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.609875664 (381h24m34.480958709s) [noise=true , address=145.79.227.32:4834 , piece_id=T6AGJLKTKW4MJTMXEBAW4YBF5BSWQRKHWYKLEU4Y5U37NAMMAJTQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.622134556 (381h24m34.493217591s) [noise=true , address=145.79.227.15:4875 , piece_id=J53SUZ5OSH7B6THTLOG6762X2VDONOODZQSPT4I3BKGOCZZV3HEA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.755361442 (381h24m34.626444487s) [noise=true , address=145.79.227.29:4943 , piece_id=DZZBLVTCTKGAHIFCHYNS5JLXQD2IBSP6XELBRGWS4QUBPYFMH7SA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.876619305 (381h24m34.74770235s) [noise=true , address=145.79.227.33:4870 , piece_id=GO6YIYMOGWGKS3KHIKAUBTOYTJLENKH6M7HJWLHC3OTNMUA4WJXA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.899187028 (381h24m34.770270073s) [noise=true , address=145.79.227.52:4445 , piece_id=SJDQEFSKBN6ERSH2WJUKMY4TEKA2XJQHDU7YBGTZWB7OQ52WE3KA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.205676652 (381h24m35.076759698s) [noise=true , address=145.79.227.42:4121 , piece_id=PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.259291003 (381h24m35.130374048s) [noise=true , address=145.79.227.48:4302 , piece_id=GIFTHYKLR2K3DJLHB4VZ6T7NMYSA5TRAKPTEEJ53UVMQTYROMECA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.326071153 (381h24m35.197154188s) [noise=true , address=45.91.116.6:4974 , piece_id=ZX3KN4IETHLW3ABY6J63I35OMTQC2ZMFKO3GBPU4PY4HDS6OA22A, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.502232413 (381h24m35.373315448s) [noise=true , address=145.79.227.47:4267 , piece_id=YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.502253854 (381h24m35.373336919s) [seg_pos={0 0}, upload=334718]: Segment reached optimal threshold of 13 pieces. Dec 21 12:53:35.502568303 (381h24m35.373651338s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 21 12:53:35.502581327 (381h24m35.373664372s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502608810 (381h24m35.373691845s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection Dec 21 12:53:35.502619169 (381h24m35.373702214s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502846553 (381h24m35.373929588s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection Dec 21 12:53:35.502853666 (381h24m35.373936701s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502873894 (381h24m35.373956929s) [seg_pos={0 0}, upload=334718]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:53:35.502965338 (381h24m35.374048383s) [upload=334718]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:53:35.502970939 (381h24m35.374053984s) [upload=334718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:53:35.794077451 (381h24m35.665160486s) [upload=334718]: Done uploading segments... Dec 21 12:53:35.794103912 (381h24m35.665186957s) [upload=334718]: Upload finished. err: <nil> Duration: 5.738650061s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.XXX.X.X..X.........X.X.X.....X..X.....X..X....X........X...] duration: 6.079073161s trace: https://jaeger.dev.storj.io/trace/5254881e4ca2c0fe FAIL (2025-12-21 12:55:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, Piece ID: YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A) context deadline exceeded; piecestore: (Node ID: 1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, Piece ID: I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ) context deadline exceeded; piecestore: (Node ID: 12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, Piece ID: 7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ) context deadline exceeded; piecestore: (Node ID: 17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, Piece ID: 36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q) context deadline exceeded ======LOGS====== ents... Dec 21 12:53:32.080943203 (381h24m31.952026248s) [upload=334718]: Got next segment. Inline: false Dec 21 12:53:32.080975975 (381h24m31.95205901s) [upload=334718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:53:32.080980384 (381h24m31.952063419s) [upload=334718]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:53:32.328092505 (381h24m32.19917554s) [seg_pos={0 0}, upload=334718]: Begin upload segment... Dec 21 12:53:32.328182436 (381h24m32.199265471s) [seg_pos={0 0}, upload=334718]: Done begin upload segment. Dec 21 12:53:32.328220579 (381h24m32.199303624s) [noise=true , address=145.79.227.48:4302 , piece_id=GIFTHYKLR2K3DJLHB4VZ6T7NMYSA5TRAKPTEEJ53UVMQTYROMECA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.331668685 (381h24m32.20275173s) [noise=true , address=145.79.227.52:4445 , piece_id=SJDQEFSKBN6ERSH2WJUKMY4TEKA2XJQHDU7YBGTZWB7OQ52WE3KA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.332500137 (381h24m32.203583172s) [noise=true , address=145.79.227.47:4267 , piece_id=YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.333277057 (381h24m32.204360092s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.334224581 (381h24m32.205307617s) [noise=true , address=145.79.227.42:4121 , piece_id=PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.334844001 (381h24m32.205927046s) [noise=true , address=45.91.116.6:4974 , piece_id=ZX3KN4IETHLW3ABY6J63I35OMTQC2ZMFKO3GBPU4PY4HDS6OA22A, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.335828375 (381h24m32.20691142s) [noise=true , address=145.79.227.29:4943 , piece_id=DZZBLVTCTKGAHIFCHYNS5JLXQD2IBSP6XELBRGWS4QUBPYFMH7SA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.336610345 (381h24m32.20769338s) [noise=true , address=145.79.227.49:4338 , piece_id=I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.337469029 (381h24m32.208552074s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.338354977 (381h24m32.209438022s) [noise=true , address=145.79.227.34:4907 , piece_id=7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.339032116 (381h24m32.210115161s) [noise=true , address=145.79.227.15:4875 , piece_id=J53SUZ5OSH7B6THTLOG6762X2VDONOODZQSPT4I3BKGOCZZV3HEA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.340261366 (381h24m32.211344401s) [noise=true , address=145.79.227.14:4839 , piece_id=RMPSFFROF52HBMWI7HYYOZRYPN3YM6CCUSZJQABGOCKQOEOMGPZA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.340406363 (381h24m32.211489398s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.341633339 (381h24m32.212716374s) [noise=true , address=145.79.227.33:4870 , piece_id=GO6YIYMOGWGKS3KHIKAUBTOYTJLENKH6M7HJWLHC3OTNMUA4WJXA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.342100158 (381h24m32.213183193s) [noise=true , address=145.79.227.41:4085 , piece_id=36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.343690196 (381h24m32.214773241s) [noise=true , address=145.79.227.32:4834 , piece_id=T6AGJLKTKW4MJTMXEBAW4YBF5BSWQRKHWYKLEU4Y5U37NAMMAJTQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:34.215235315 (381h24m34.08631835s) [upload=334718]: Next returned nil segment Dec 21 12:53:34.215255653 (381h24m34.086338689s) [upload=334718]: Waiting for error group managing segments... Dec 21 12:53:34.523622594 (381h24m34.394705629s) [noise=true , address=145.79.227.49:4338 , piece_id=I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.525892566 (381h24m34.396975601s) [noise=true , address=145.79.227.14:4839 , piece_id=RMPSFFROF52HBMWI7HYYOZRYPN3YM6CCUSZJQABGOCKQOEOMGPZA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.535347309 (381h24m34.406430344s) [noise=true , address=145.79.227.41:4085 , piece_id=36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.592006026 (381h24m34.463089061s) [noise=true , address=145.79.227.34:4907 , piece_id=7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.609875664 (381h24m34.480958709s) [noise=true , address=145.79.227.32:4834 , piece_id=T6AGJLKTKW4MJTMXEBAW4YBF5BSWQRKHWYKLEU4Y5U37NAMMAJTQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.622134556 (381h24m34.493217591s) [noise=true , address=145.79.227.15:4875 , piece_id=J53SUZ5OSH7B6THTLOG6762X2VDONOODZQSPT4I3BKGOCZZV3HEA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.755361442 (381h24m34.626444487s) [noise=true , address=145.79.227.29:4943 , piece_id=DZZBLVTCTKGAHIFCHYNS5JLXQD2IBSP6XELBRGWS4QUBPYFMH7SA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.876619305 (381h24m34.74770235s) [noise=true , address=145.79.227.33:4870 , piece_id=GO6YIYMOGWGKS3KHIKAUBTOYTJLENKH6M7HJWLHC3OTNMUA4WJXA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.899187028 (381h24m34.770270073s) [noise=true , address=145.79.227.52:4445 , piece_id=SJDQEFSKBN6ERSH2WJUKMY4TEKA2XJQHDU7YBGTZWB7OQ52WE3KA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.205676652 (381h24m35.076759698s) [noise=true , address=145.79.227.42:4121 , piece_id=PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.259291003 (381h24m35.130374048s) [noise=true , address=145.79.227.48:4302 , piece_id=GIFTHYKLR2K3DJLHB4VZ6T7NMYSA5TRAKPTEEJ53UVMQTYROMECA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.326071153 (381h24m35.197154188s) [noise=true , address=45.91.116.6:4974 , piece_id=ZX3KN4IETHLW3ABY6J63I35OMTQC2ZMFKO3GBPU4PY4HDS6OA22A, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.502232413 (381h24m35.373315448s) [noise=true , address=145.79.227.47:4267 , piece_id=YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.502253854 (381h24m35.373336919s) [seg_pos={0 0}, upload=334718]: Segment reached optimal threshold of 13 pieces. Dec 21 12:53:35.502568303 (381h24m35.373651338s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 21 12:53:35.502581327 (381h24m35.373664372s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502608810 (381h24m35.373691845s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection Dec 21 12:53:35.502619169 (381h24m35.373702214s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502846553 (381h24m35.373929588s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection Dec 21 12:53:35.502853666 (381h24m35.373936701s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502873894 (381h24m35.373956929s) [seg_pos={0 0}, upload=334718]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:53:35.502965338 (381h24m35.374048383s) [upload=334718]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:53:35.502970939 (381h24m35.374053984s) [upload=334718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:53:35.794077451 (381h24m35.665160486s) [upload=334718]: Done uploading segments... Dec 21 12:53:35.794103912 (381h24m35.665186957s) [upload=334718]: Upload finished. err: <nil> Duration: 6.07909863s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X.XXX.X.X..X.........X.X.X.....X..X.....X..X....X........X..] duration: 5.815701986s trace: https://jaeger.dev.storj.io/trace/5254881e4ca2c0fe FAIL (2025-12-21 12:55:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, Piece ID: YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A) context deadline exceeded; piecestore: (Node ID: 1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, Piece ID: I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ) context deadline exceeded; piecestore: (Node ID: 12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, Piece ID: 7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ) context deadline exceeded; piecestore: (Node ID: 17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, Piece ID: 36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q) context deadline exceeded ======LOGS====== ents... Dec 21 12:53:32.080943203 (381h24m31.952026248s) [upload=334718]: Got next segment. Inline: false Dec 21 12:53:32.080975975 (381h24m31.95205901s) [upload=334718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:53:32.080980384 (381h24m31.952063419s) [upload=334718]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:53:32.328092505 (381h24m32.19917554s) [seg_pos={0 0}, upload=334718]: Begin upload segment... Dec 21 12:53:32.328182436 (381h24m32.199265471s) [seg_pos={0 0}, upload=334718]: Done begin upload segment. Dec 21 12:53:32.328220579 (381h24m32.199303624s) [noise=true , address=145.79.227.48:4302 , piece_id=GIFTHYKLR2K3DJLHB4VZ6T7NMYSA5TRAKPTEEJ53UVMQTYROMECA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.331668685 (381h24m32.20275173s) [noise=true , address=145.79.227.52:4445 , piece_id=SJDQEFSKBN6ERSH2WJUKMY4TEKA2XJQHDU7YBGTZWB7OQ52WE3KA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.332500137 (381h24m32.203583172s) [noise=true , address=145.79.227.47:4267 , piece_id=YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.333277057 (381h24m32.204360092s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.334224581 (381h24m32.205307617s) [noise=true , address=145.79.227.42:4121 , piece_id=PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.334844001 (381h24m32.205927046s) [noise=true , address=45.91.116.6:4974 , piece_id=ZX3KN4IETHLW3ABY6J63I35OMTQC2ZMFKO3GBPU4PY4HDS6OA22A, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.335828375 (381h24m32.20691142s) [noise=true , address=145.79.227.29:4943 , piece_id=DZZBLVTCTKGAHIFCHYNS5JLXQD2IBSP6XELBRGWS4QUBPYFMH7SA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.336610345 (381h24m32.20769338s) [noise=true , address=145.79.227.49:4338 , piece_id=I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.337469029 (381h24m32.208552074s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.338354977 (381h24m32.209438022s) [noise=true , address=145.79.227.34:4907 , piece_id=7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.339032116 (381h24m32.210115161s) [noise=true , address=145.79.227.15:4875 , piece_id=J53SUZ5OSH7B6THTLOG6762X2VDONOODZQSPT4I3BKGOCZZV3HEA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.340261366 (381h24m32.211344401s) [noise=true , address=145.79.227.14:4839 , piece_id=RMPSFFROF52HBMWI7HYYOZRYPN3YM6CCUSZJQABGOCKQOEOMGPZA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.340406363 (381h24m32.211489398s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.341633339 (381h24m32.212716374s) [noise=true , address=145.79.227.33:4870 , piece_id=GO6YIYMOGWGKS3KHIKAUBTOYTJLENKH6M7HJWLHC3OTNMUA4WJXA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.342100158 (381h24m32.213183193s) [noise=true , address=145.79.227.41:4085 , piece_id=36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.343690196 (381h24m32.214773241s) [noise=true , address=145.79.227.32:4834 , piece_id=T6AGJLKTKW4MJTMXEBAW4YBF5BSWQRKHWYKLEU4Y5U37NAMMAJTQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:34.215235315 (381h24m34.08631835s) [upload=334718]: Next returned nil segment Dec 21 12:53:34.215255653 (381h24m34.086338689s) [upload=334718]: Waiting for error group managing segments... Dec 21 12:53:34.523622594 (381h24m34.394705629s) [noise=true , address=145.79.227.49:4338 , piece_id=I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.525892566 (381h24m34.396975601s) [noise=true , address=145.79.227.14:4839 , piece_id=RMPSFFROF52HBMWI7HYYOZRYPN3YM6CCUSZJQABGOCKQOEOMGPZA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.535347309 (381h24m34.406430344s) [noise=true , address=145.79.227.41:4085 , piece_id=36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.592006026 (381h24m34.463089061s) [noise=true , address=145.79.227.34:4907 , piece_id=7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.609875664 (381h24m34.480958709s) [noise=true , address=145.79.227.32:4834 , piece_id=T6AGJLKTKW4MJTMXEBAW4YBF5BSWQRKHWYKLEU4Y5U37NAMMAJTQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.622134556 (381h24m34.493217591s) [noise=true , address=145.79.227.15:4875 , piece_id=J53SUZ5OSH7B6THTLOG6762X2VDONOODZQSPT4I3BKGOCZZV3HEA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.755361442 (381h24m34.626444487s) [noise=true , address=145.79.227.29:4943 , piece_id=DZZBLVTCTKGAHIFCHYNS5JLXQD2IBSP6XELBRGWS4QUBPYFMH7SA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.876619305 (381h24m34.74770235s) [noise=true , address=145.79.227.33:4870 , piece_id=GO6YIYMOGWGKS3KHIKAUBTOYTJLENKH6M7HJWLHC3OTNMUA4WJXA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.899187028 (381h24m34.770270073s) [noise=true , address=145.79.227.52:4445 , piece_id=SJDQEFSKBN6ERSH2WJUKMY4TEKA2XJQHDU7YBGTZWB7OQ52WE3KA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.205676652 (381h24m35.076759698s) [noise=true , address=145.79.227.42:4121 , piece_id=PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.259291003 (381h24m35.130374048s) [noise=true , address=145.79.227.48:4302 , piece_id=GIFTHYKLR2K3DJLHB4VZ6T7NMYSA5TRAKPTEEJ53UVMQTYROMECA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.326071153 (381h24m35.197154188s) [noise=true , address=45.91.116.6:4974 , piece_id=ZX3KN4IETHLW3ABY6J63I35OMTQC2ZMFKO3GBPU4PY4HDS6OA22A, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.502232413 (381h24m35.373315448s) [noise=true , address=145.79.227.47:4267 , piece_id=YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.502253854 (381h24m35.373336919s) [seg_pos={0 0}, upload=334718]: Segment reached optimal threshold of 13 pieces. Dec 21 12:53:35.502568303 (381h24m35.373651338s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 21 12:53:35.502581327 (381h24m35.373664372s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502608810 (381h24m35.373691845s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection Dec 21 12:53:35.502619169 (381h24m35.373702214s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502846553 (381h24m35.373929588s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection Dec 21 12:53:35.502853666 (381h24m35.373936701s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502873894 (381h24m35.373956929s) [seg_pos={0 0}, upload=334718]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:53:35.502965338 (381h24m35.374048383s) [upload=334718]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:53:35.502970939 (381h24m35.374053984s) [upload=334718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:53:35.794077451 (381h24m35.665160486s) [upload=334718]: Done uploading segments... Dec 21 12:53:35.794103912 (381h24m35.665186957s) [upload=334718]: Upload finished. err: <nil> Duration: 5.81573517s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X.XXX.X.X..X.........X.X.X.....X..X.....X..X....X........X.] duration: 5.692130351s trace: https://jaeger.dev.storj.io/trace/5254881e4ca2c0fe FAIL (2025-12-21 12:54:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, Piece ID: YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ) context deadline exceeded; piecestore: (Node ID: 1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, Piece ID: PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A) context deadline exceeded; piecestore: (Node ID: 1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, Piece ID: I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ) context deadline exceeded; piecestore: (Node ID: 12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, Piece ID: 7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ) context deadline exceeded; piecestore: (Node ID: 17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, Piece ID: 36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q) context deadline exceeded ======LOGS====== ents... Dec 21 12:53:32.080943203 (381h24m31.952026248s) [upload=334718]: Got next segment. Inline: false Dec 21 12:53:32.080975975 (381h24m31.95205901s) [upload=334718]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 12:53:32.080980384 (381h24m31.952063419s) [upload=334718]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 12:53:32.328092505 (381h24m32.19917554s) [seg_pos={0 0}, upload=334718]: Begin upload segment... Dec 21 12:53:32.328182436 (381h24m32.199265471s) [seg_pos={0 0}, upload=334718]: Done begin upload segment. Dec 21 12:53:32.328220579 (381h24m32.199303624s) [noise=true , address=145.79.227.48:4302 , piece_id=GIFTHYKLR2K3DJLHB4VZ6T7NMYSA5TRAKPTEEJ53UVMQTYROMECA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.331668685 (381h24m32.20275173s) [noise=true , address=145.79.227.52:4445 , piece_id=SJDQEFSKBN6ERSH2WJUKMY4TEKA2XJQHDU7YBGTZWB7OQ52WE3KA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.332500137 (381h24m32.203583172s) [noise=true , address=145.79.227.47:4267 , piece_id=YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.333277057 (381h24m32.204360092s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.334224581 (381h24m32.205307617s) [noise=true , address=145.79.227.42:4121 , piece_id=PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.334844001 (381h24m32.205927046s) [noise=true , address=45.91.116.6:4974 , piece_id=ZX3KN4IETHLW3ABY6J63I35OMTQC2ZMFKO3GBPU4PY4HDS6OA22A, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.335828375 (381h24m32.20691142s) [noise=true , address=145.79.227.29:4943 , piece_id=DZZBLVTCTKGAHIFCHYNS5JLXQD2IBSP6XELBRGWS4QUBPYFMH7SA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.336610345 (381h24m32.20769338s) [noise=true , address=145.79.227.49:4338 , piece_id=I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.337469029 (381h24m32.208552074s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.338354977 (381h24m32.209438022s) [noise=true , address=145.79.227.34:4907 , piece_id=7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.339032116 (381h24m32.210115161s) [noise=true , address=145.79.227.15:4875 , piece_id=J53SUZ5OSH7B6THTLOG6762X2VDONOODZQSPT4I3BKGOCZZV3HEA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.340261366 (381h24m32.211344401s) [noise=true , address=145.79.227.14:4839 , piece_id=RMPSFFROF52HBMWI7HYYOZRYPN3YM6CCUSZJQABGOCKQOEOMGPZA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.340406363 (381h24m32.211489398s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.341633339 (381h24m32.212716374s) [noise=true , address=145.79.227.33:4870 , piece_id=GO6YIYMOGWGKS3KHIKAUBTOYTJLENKH6M7HJWLHC3OTNMUA4WJXA, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.342100158 (381h24m32.213183193s) [noise=true , address=145.79.227.41:4085 , piece_id=36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:32.343690196 (381h24m32.214773241s) [noise=true , address=145.79.227.32:4834 , piece_id=T6AGJLKTKW4MJTMXEBAW4YBF5BSWQRKHWYKLEU4Y5U37NAMMAJTQ, seg_pos={0 0}, upload=334718]: Uploading piece... Dec 21 12:53:34.215235315 (381h24m34.08631835s) [upload=334718]: Next returned nil segment Dec 21 12:53:34.215255653 (381h24m34.086338689s) [upload=334718]: Waiting for error group managing segments... Dec 21 12:53:34.523622594 (381h24m34.394705629s) [noise=true , address=145.79.227.49:4338 , piece_id=I4TLOS6LO27H627CBXANWSQWZK7Y2KFYP2QDDFBIJ6ES2JW54VKQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.525892566 (381h24m34.396975601s) [noise=true , address=145.79.227.14:4839 , piece_id=RMPSFFROF52HBMWI7HYYOZRYPN3YM6CCUSZJQABGOCKQOEOMGPZA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.535347309 (381h24m34.406430344s) [noise=true , address=145.79.227.41:4085 , piece_id=36MNJR5J7GPZFMBPQ7IFTYGS4N7Q7WE7LU2GSTVODX663NZDSF6Q, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.592006026 (381h24m34.463089061s) [noise=true , address=145.79.227.34:4907 , piece_id=7T4V6MEXMFETGKTLJ5CECDZGXLK5UNLLG7MOELHOW7XQ7PJSQ3AQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.609875664 (381h24m34.480958709s) [noise=true , address=145.79.227.32:4834 , piece_id=T6AGJLKTKW4MJTMXEBAW4YBF5BSWQRKHWYKLEU4Y5U37NAMMAJTQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.622134556 (381h24m34.493217591s) [noise=true , address=145.79.227.15:4875 , piece_id=J53SUZ5OSH7B6THTLOG6762X2VDONOODZQSPT4I3BKGOCZZV3HEA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.755361442 (381h24m34.626444487s) [noise=true , address=145.79.227.29:4943 , piece_id=DZZBLVTCTKGAHIFCHYNS5JLXQD2IBSP6XELBRGWS4QUBPYFMH7SA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.876619305 (381h24m34.74770235s) [noise=true , address=145.79.227.33:4870 , piece_id=GO6YIYMOGWGKS3KHIKAUBTOYTJLENKH6M7HJWLHC3OTNMUA4WJXA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:34.899187028 (381h24m34.770270073s) [noise=true , address=145.79.227.52:4445 , piece_id=SJDQEFSKBN6ERSH2WJUKMY4TEKA2XJQHDU7YBGTZWB7OQ52WE3KA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.205676652 (381h24m35.076759698s) [noise=true , address=145.79.227.42:4121 , piece_id=PV7XHYXGFQIBX4VUZWY3TLTYDFTRVXAVIT3PLDSKZASS4EEWJU2A, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.259291003 (381h24m35.130374048s) [noise=true , address=145.79.227.48:4302 , piece_id=GIFTHYKLR2K3DJLHB4VZ6T7NMYSA5TRAKPTEEJ53UVMQTYROMECA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.326071153 (381h24m35.197154188s) [noise=true , address=45.91.116.6:4974 , piece_id=ZX3KN4IETHLW3ABY6J63I35OMTQC2ZMFKO3GBPU4PY4HDS6OA22A, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.502232413 (381h24m35.373315448s) [noise=true , address=145.79.227.47:4267 , piece_id=YE4NZ2VKHLHBF6DYZ7IYVEUHL7Q5KRQOXLXD5HXZC4JNITMU6VYQ, seg_pos={0 0}, upload=334718]: Done uploading piece. err: <nil> Dec 21 12:53:35.502253854 (381h24m35.373336919s) [seg_pos={0 0}, upload=334718]: Segment reached optimal threshold of 13 pieces. Dec 21 12:53:35.502568303 (381h24m35.373651338s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: expected piece hash; context canceled; EOF Dec 21 12:53:35.502581327 (381h24m35.373664372s) [noise=true , address=145.79.227.39:4015 , piece_id=LJB22XSMKS7PJHQJZ32MVGBSIJOC7EVGYFDXUXQKPHE7YGJVZGNA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502608810 (381h24m35.373691845s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy): protocol: write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52108->145.79.227.9:4366: use of closed network connection Dec 21 12:53:35.502619169 (381h24m35.373702214s) [noise=true , address=145.79.227.9:4366 , piece_id=HUFEKV5NM4IICP3Z7TDJ7HBM3BKIKLVIMR4GGSZNAIMIKJ7GFIUA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502846553 (381h24m35.373929588s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34842->145.79.227.37:4567: use of closed network connection Dec 21 12:53:35.502853666 (381h24m35.373936701s) [noise=true , address=145.79.227.37:4567 , piece_id=ECSXNMGRJ2UDRTLWJB4YSOBIMC6O67KJM4Y7ZJMS4OIHZ7MCZXVA, seg_pos={0 0}, upload=334718]: Long tail cancellation due to optimal threshold Dec 21 12:53:35.502873894 (381h24m35.373956929s) [seg_pos={0 0}, upload=334718]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 12:53:35.502965338 (381h24m35.374048383s) [upload=334718]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 12:53:35.502970939 (381h24m35.374053984s) [upload=334718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 12:53:35.794077451 (381h24m35.665160486s) [upload=334718]: Done uploading segments... Dec 21 12:53:35.794103912 (381h24m35.665186957s) [upload=334718]: Upload finished. err: <nil> Duration: 5.692156852s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)