Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X..X.....XX.X.....X.....X.................X...X.X.X...X.....] duration: 6.141567118s trace: https://jaeger.dev.storj.io/trace/49e32ea040e6dc54 FAIL (2025-12-30 20:03:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: UAV7EMD24JN2CVJ3WG5C5HDYHBTCW4TYLGC35QZI35A7EKIMDVGA) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VLIHBJ7BBCKC7FC53TNNOWDXLWI5C7FOJ3RPHCXPTZM7JVI3P7PA) context deadline exceeded ======LOGS====== Dec 30 19:59:22.789859524 (604h30m22.660942559s) [upload=529831]: Starting upload Dec 30 19:59:22.789937492 (604h30m22.661020537s) [upload=529831]: Done starting upload Dec 30 19:59:22.790028305 (604h30m22.66111134s) [upload=529831]: Uploading segments... Dec 30 19:59:22.790207306 (604h30m22.661290341s) [upload=529831]: Got next segment. Inline: false Dec 30 19:59:22.790263323 (604h30m22.661346378s) [upload=529831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 19:59:22.790269495 (604h30m22.66135254s) [upload=529831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 19:59:23.044447967 (604h30m22.915531042s) [seg_pos={0 0}, upload=529831]: Begin upload segment... Dec 30 19:59:23.044503883 (604h30m22.915586938s) [seg_pos={0 0}, upload=529831]: Done begin upload segment. Dec 30 19:59:23.044537507 (604h30m22.915620543s) [noise=true , address=145.79.227.38:4978 , piece_id=XAWLFWG3GHGTY2WVIJDMLF3QY27LNYQFMGDLNK3HCJZPQSTBXY3A, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.044594004 (604h30m22.915677039s) [noise=true , address=45.91.116.6:4975 , piece_id=QNI5FNAKGYEOP3HDSI53FU4AQPV6PUQCK4SGBXCQRL3AJCZBUXPA, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.045080781 (604h30m22.916163816s) [noise=true , address=77.90.31.10:4517 , piece_id=5V4O7IOLKG7SMJDY6RCWGDJH222XCXTNEZ5YQO2ET5ZQQTU6LB3Q, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.045471716 (604h30m22.916554751s) [noise=true , address=145.79.227.13:4802 , piece_id=J3PGR67GK33SG2CTVCWN7AZDWUTMZ577ZRFWLSVKB3657CHJUMIQ, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.045918206 (604h30m22.917001251s) [noise=true , address=45.91.116.11:4117 , piece_id=VLIHBJ7BBCKC7FC53TNNOWDXLWI5C7FOJ3RPHCXPTZM7JVI3P7PA, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.046449959 (604h30m22.917533004s) [noise=true , address=145.79.227.31:4797 , piece_id=RKCEF2OC2MXOU3CQURIWIVGEAAJFKIHF43H3UPKICRDIUK4OUH4A, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.046883935 (604h30m22.91796698s) [noise=true , address=145.79.227.49:4338 , piece_id=OYMRWST4K5KFB5PV3HZTOTK2JCV7MPZWEZBVZDHVRM7NJZBLT2NA, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.047737561 (604h30m22.918820606s) [noise=true , address=145.79.227.42:4121 , piece_id=ECWXPU2NM6HHSFFUZJAHMZPIZL6GN3WKFPBGCGOLCLZOFCZPD5KQ, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.049760984 (604h30m22.920844029s) [noise=true , address=145.79.227.12:4765 , piece_id=FCRGW3R3T2H5OOCTXGAUL5WWEX5C7O4LQR5LGYSRMAGDD7CIQW4Q, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.049887454 (604h30m22.920970499s) [noise=true , address=45.91.116.10:4081 , piece_id=5OH3T6YWTPNWNAJTGNQIG4TPMWFH646OTTCQDUMYAZUHDT3HQM4A, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.053052571 (604h30m22.924135616s) [noise=true , address=145.79.227.11:4731 , piece_id=XUZBIIS3CB73NCCLURSX2Z4DUCQ6L2EWCE53I4XB62KOUSDYWAIA, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.055404520 (604h30m22.926487565s) [noise=true , address=145.79.227.45:4229 , piece_id=BB6HAB7WGPNLXIZCQTNDHNNXMG2VGHCMGVPSPV7DDYMQIFUC4GLA, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.059215657 (604h30m22.930298692s) [noise=true , address=77.90.31.21:4878 , piece_id=CALPIEYPGBYEVEGBEMM4TVCBSAT3ELHL3S7EDMW3EMSEDKOCBVXA, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.063539760 (604h30m22.934622805s) [noise=true , address=145.79.227.30:4762 , piece_id=4SRJQBHND2IXLDZSS3ZH7KKYSKBTMNIYEGFF7742NERNJYJYALVQ, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.064363259 (604h30m22.935446294s) [noise=true , address=145.79.227.4:4185 , piece_id=OQIBCFJYWU5J7VHH2GXOPX3PVLLIRMNFZSCUMYNOLKLTYKGY5QIQ, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.065657292 (604h30m22.936740338s) [noise=true , address=45.91.116.8:4048 , piece_id=UAV7EMD24JN2CVJ3WG5C5HDYHBTCW4TYLGC35QZI35A7EKIMDVGA, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:24.746124353 (604h30m24.617207388s) [upload=529831]: Next returned nil segment Dec 30 19:59:24.746141626 (604h30m24.617224661s) [upload=529831]: Waiting for error group managing segments... Dec 30 19:59:25.024653914 (604h30m24.895736949s) [noise=true , address=145.79.227.13:4802 , piece_id=J3PGR67GK33SG2CTVCWN7AZDWUTMZ577ZRFWLSVKB3657CHJUMIQ, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.035276661 (604h30m24.906359696s) [noise=true , address=145.79.227.38:4978 , piece_id=XAWLFWG3GHGTY2WVIJDMLF3QY27LNYQFMGDLNK3HCJZPQSTBXY3A, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.043199637 (604h30m24.914282672s) [noise=true , address=145.79.227.4:4185 , piece_id=OQIBCFJYWU5J7VHH2GXOPX3PVLLIRMNFZSCUMYNOLKLTYKGY5QIQ, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.054826626 (604h30m24.925909661s) [noise=true , address=145.79.227.42:4121 , piece_id=ECWXPU2NM6HHSFFUZJAHMZPIZL6GN3WKFPBGCGOLCLZOFCZPD5KQ, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.064076520 (604h30m24.935159565s) [noise=true , address=145.79.227.12:4765 , piece_id=FCRGW3R3T2H5OOCTXGAUL5WWEX5C7O4LQR5LGYSRMAGDD7CIQW4Q, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.067094476 (604h30m24.938177511s) [noise=true , address=145.79.227.31:4797 , piece_id=RKCEF2OC2MXOU3CQURIWIVGEAAJFKIHF43H3UPKICRDIUK4OUH4A, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.134512319 (604h30m25.005595354s) [noise=true , address=145.79.227.11:4731 , piece_id=XUZBIIS3CB73NCCLURSX2Z4DUCQ6L2EWCE53I4XB62KOUSDYWAIA, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.137865084 (604h30m25.008948119s) [noise=true , address=145.79.227.45:4229 , piece_id=BB6HAB7WGPNLXIZCQTNDHNNXMG2VGHCMGVPSPV7DDYMQIFUC4GLA, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.206505284 (604h30m25.077588329s) [noise=true , address=77.90.31.10:4517 , piece_id=5V4O7IOLKG7SMJDY6RCWGDJH222XCXTNEZ5YQO2ET5ZQQTU6LB3Q, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.347801863 (604h30m25.218884898s) [noise=true , address=45.91.116.8:4048 , piece_id=UAV7EMD24JN2CVJ3WG5C5HDYHBTCW4TYLGC35QZI35A7EKIMDVGA, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.373575631 (604h30m25.244658666s) [noise=true , address=45.91.116.6:4975 , piece_id=QNI5FNAKGYEOP3HDSI53FU4AQPV6PUQCK4SGBXCQRL3AJCZBUXPA, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.438579707 (604h30m25.309662752s) [noise=true , address=45.91.116.11:4117 , piece_id=VLIHBJ7BBCKC7FC53TNNOWDXLWI5C7FOJ3RPHCXPTZM7JVI3P7PA, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:26.006186322 (604h30m25.877269367s) [noise=true , address=77.90.31.21:4878 , piece_id=CALPIEYPGBYEVEGBEMM4TVCBSAT3ELHL3S7EDMW3EMSEDKOCBVXA, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:26.006212882 (604h30m25.877295918s) [seg_pos={0 0}, upload=529831]: Segment reached optimal threshold of 13 pieces. Dec 30 19:59:26.006585312 (604h30m25.877668387s) [noise=true , address=45.91.116.10:4081 , piece_id=5OH3T6YWTPNWNAJTGNQIG4TPMWFH646OTTCQDUMYAZUHDT3HQM4A, seg_pos={0 0}, upload=529831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 30 19:59:26.006597174 (604h30m25.877680249s) [noise=true , address=45.91.116.10:4081 , piece_id=5OH3T6YWTPNWNAJTGNQIG4TPMWFH646OTTCQDUMYAZUHDT3HQM4A, seg_pos={0 0}, upload=529831]: Long tail cancellation due to optimal threshold Dec 30 19:59:26.006651678 (604h30m25.877734713s) [noise=true , address=145.79.227.49:4338 , piece_id=OYMRWST4K5KFB5PV3HZTOTK2JCV7MPZWEZBVZDHVRM7NJZBLT2NA, seg_pos={0 0}, upload=529831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 30 19:59:26.006659393 (604h30m25.877742428s) [noise=true , address=145.79.227.49:4338 , piece_id=OYMRWST4K5KFB5PV3HZTOTK2JCV7MPZWEZBVZDHVRM7NJZBLT2NA, seg_pos={0 0}, upload=529831]: Long tail cancellation due to optimal threshold Dec 30 19:59:26.006668790 (604h30m25.877751826s) [noise=true , address=145.79.227.30:4762 , piece_id=4SRJQBHND2IXLDZSS3ZH7KKYSKBTMNIYEGFF7742NERNJYJYALVQ, seg_pos={0 0}, upload=529831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 30 19:59:26.006673119 (604h30m25.877756154s) [noise=true , address=145.79.227.30:4762 , piece_id=4SRJQBHND2IXLDZSS3ZH7KKYSKBTMNIYEGFF7742NERNJYJYALVQ, seg_pos={0 0}, upload=529831]: Long tail cancellation due to optimal threshold Dec 30 19:59:26.006690051 (604h30m25.877773086s) [seg_pos={0 0}, upload=529831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 19:59:26.006786083 (604h30m25.877869119s) [upload=529831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:59:26.006791164 (604h30m25.877874209s) [upload=529831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:59:26.299587881 (604h30m26.170670916s) [upload=529831]: Done uploading segments... Dec 30 19:59:26.299606246 (604h30m26.170689291s) [upload=529831]: Upload finished. err: <nil> Duration: 6.141594401s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.X..X.....XX.X.....X.....X.................X...X.X.X...X....] duration: 6.022536265s trace: https://jaeger.dev.storj.io/trace/49e32ea040e6dc54 FAIL (2025-12-30 20:02:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: UAV7EMD24JN2CVJ3WG5C5HDYHBTCW4TYLGC35QZI35A7EKIMDVGA) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VLIHBJ7BBCKC7FC53TNNOWDXLWI5C7FOJ3RPHCXPTZM7JVI3P7PA) context deadline exceeded ======LOGS====== Dec 30 19:59:22.789859524 (604h30m22.660942559s) [upload=529831]: Starting upload Dec 30 19:59:22.789937492 (604h30m22.661020537s) [upload=529831]: Done starting upload Dec 30 19:59:22.790028305 (604h30m22.66111134s) [upload=529831]: Uploading segments... Dec 30 19:59:22.790207306 (604h30m22.661290341s) [upload=529831]: Got next segment. Inline: false Dec 30 19:59:22.790263323 (604h30m22.661346378s) [upload=529831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 19:59:22.790269495 (604h30m22.66135254s) [upload=529831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 19:59:23.044447967 (604h30m22.915531042s) [seg_pos={0 0}, upload=529831]: Begin upload segment... Dec 30 19:59:23.044503883 (604h30m22.915586938s) [seg_pos={0 0}, upload=529831]: Done begin upload segment. Dec 30 19:59:23.044537507 (604h30m22.915620543s) [noise=true , address=145.79.227.38:4978 , piece_id=XAWLFWG3GHGTY2WVIJDMLF3QY27LNYQFMGDLNK3HCJZPQSTBXY3A, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.044594004 (604h30m22.915677039s) [noise=true , address=45.91.116.6:4975 , piece_id=QNI5FNAKGYEOP3HDSI53FU4AQPV6PUQCK4SGBXCQRL3AJCZBUXPA, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.045080781 (604h30m22.916163816s) [noise=true , address=77.90.31.10:4517 , piece_id=5V4O7IOLKG7SMJDY6RCWGDJH222XCXTNEZ5YQO2ET5ZQQTU6LB3Q, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.045471716 (604h30m22.916554751s) [noise=true , address=145.79.227.13:4802 , piece_id=J3PGR67GK33SG2CTVCWN7AZDWUTMZ577ZRFWLSVKB3657CHJUMIQ, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.045918206 (604h30m22.917001251s) [noise=true , address=45.91.116.11:4117 , piece_id=VLIHBJ7BBCKC7FC53TNNOWDXLWI5C7FOJ3RPHCXPTZM7JVI3P7PA, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.046449959 (604h30m22.917533004s) [noise=true , address=145.79.227.31:4797 , piece_id=RKCEF2OC2MXOU3CQURIWIVGEAAJFKIHF43H3UPKICRDIUK4OUH4A, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.046883935 (604h30m22.91796698s) [noise=true , address=145.79.227.49:4338 , piece_id=OYMRWST4K5KFB5PV3HZTOTK2JCV7MPZWEZBVZDHVRM7NJZBLT2NA, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.047737561 (604h30m22.918820606s) [noise=true , address=145.79.227.42:4121 , piece_id=ECWXPU2NM6HHSFFUZJAHMZPIZL6GN3WKFPBGCGOLCLZOFCZPD5KQ, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.049760984 (604h30m22.920844029s) [noise=true , address=145.79.227.12:4765 , piece_id=FCRGW3R3T2H5OOCTXGAUL5WWEX5C7O4LQR5LGYSRMAGDD7CIQW4Q, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.049887454 (604h30m22.920970499s) [noise=true , address=45.91.116.10:4081 , piece_id=5OH3T6YWTPNWNAJTGNQIG4TPMWFH646OTTCQDUMYAZUHDT3HQM4A, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.053052571 (604h30m22.924135616s) [noise=true , address=145.79.227.11:4731 , piece_id=XUZBIIS3CB73NCCLURSX2Z4DUCQ6L2EWCE53I4XB62KOUSDYWAIA, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.055404520 (604h30m22.926487565s) [noise=true , address=145.79.227.45:4229 , piece_id=BB6HAB7WGPNLXIZCQTNDHNNXMG2VGHCMGVPSPV7DDYMQIFUC4GLA, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.059215657 (604h30m22.930298692s) [noise=true , address=77.90.31.21:4878 , piece_id=CALPIEYPGBYEVEGBEMM4TVCBSAT3ELHL3S7EDMW3EMSEDKOCBVXA, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.063539760 (604h30m22.934622805s) [noise=true , address=145.79.227.30:4762 , piece_id=4SRJQBHND2IXLDZSS3ZH7KKYSKBTMNIYEGFF7742NERNJYJYALVQ, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.064363259 (604h30m22.935446294s) [noise=true , address=145.79.227.4:4185 , piece_id=OQIBCFJYWU5J7VHH2GXOPX3PVLLIRMNFZSCUMYNOLKLTYKGY5QIQ, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.065657292 (604h30m22.936740338s) [noise=true , address=45.91.116.8:4048 , piece_id=UAV7EMD24JN2CVJ3WG5C5HDYHBTCW4TYLGC35QZI35A7EKIMDVGA, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:24.746124353 (604h30m24.617207388s) [upload=529831]: Next returned nil segment Dec 30 19:59:24.746141626 (604h30m24.617224661s) [upload=529831]: Waiting for error group managing segments... Dec 30 19:59:25.024653914 (604h30m24.895736949s) [noise=true , address=145.79.227.13:4802 , piece_id=J3PGR67GK33SG2CTVCWN7AZDWUTMZ577ZRFWLSVKB3657CHJUMIQ, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.035276661 (604h30m24.906359696s) [noise=true , address=145.79.227.38:4978 , piece_id=XAWLFWG3GHGTY2WVIJDMLF3QY27LNYQFMGDLNK3HCJZPQSTBXY3A, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.043199637 (604h30m24.914282672s) [noise=true , address=145.79.227.4:4185 , piece_id=OQIBCFJYWU5J7VHH2GXOPX3PVLLIRMNFZSCUMYNOLKLTYKGY5QIQ, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.054826626 (604h30m24.925909661s) [noise=true , address=145.79.227.42:4121 , piece_id=ECWXPU2NM6HHSFFUZJAHMZPIZL6GN3WKFPBGCGOLCLZOFCZPD5KQ, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.064076520 (604h30m24.935159565s) [noise=true , address=145.79.227.12:4765 , piece_id=FCRGW3R3T2H5OOCTXGAUL5WWEX5C7O4LQR5LGYSRMAGDD7CIQW4Q, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.067094476 (604h30m24.938177511s) [noise=true , address=145.79.227.31:4797 , piece_id=RKCEF2OC2MXOU3CQURIWIVGEAAJFKIHF43H3UPKICRDIUK4OUH4A, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.134512319 (604h30m25.005595354s) [noise=true , address=145.79.227.11:4731 , piece_id=XUZBIIS3CB73NCCLURSX2Z4DUCQ6L2EWCE53I4XB62KOUSDYWAIA, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.137865084 (604h30m25.008948119s) [noise=true , address=145.79.227.45:4229 , piece_id=BB6HAB7WGPNLXIZCQTNDHNNXMG2VGHCMGVPSPV7DDYMQIFUC4GLA, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.206505284 (604h30m25.077588329s) [noise=true , address=77.90.31.10:4517 , piece_id=5V4O7IOLKG7SMJDY6RCWGDJH222XCXTNEZ5YQO2ET5ZQQTU6LB3Q, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.347801863 (604h30m25.218884898s) [noise=true , address=45.91.116.8:4048 , piece_id=UAV7EMD24JN2CVJ3WG5C5HDYHBTCW4TYLGC35QZI35A7EKIMDVGA, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.373575631 (604h30m25.244658666s) [noise=true , address=45.91.116.6:4975 , piece_id=QNI5FNAKGYEOP3HDSI53FU4AQPV6PUQCK4SGBXCQRL3AJCZBUXPA, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.438579707 (604h30m25.309662752s) [noise=true , address=45.91.116.11:4117 , piece_id=VLIHBJ7BBCKC7FC53TNNOWDXLWI5C7FOJ3RPHCXPTZM7JVI3P7PA, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:26.006186322 (604h30m25.877269367s) [noise=true , address=77.90.31.21:4878 , piece_id=CALPIEYPGBYEVEGBEMM4TVCBSAT3ELHL3S7EDMW3EMSEDKOCBVXA, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:26.006212882 (604h30m25.877295918s) [seg_pos={0 0}, upload=529831]: Segment reached optimal threshold of 13 pieces. Dec 30 19:59:26.006585312 (604h30m25.877668387s) [noise=true , address=45.91.116.10:4081 , piece_id=5OH3T6YWTPNWNAJTGNQIG4TPMWFH646OTTCQDUMYAZUHDT3HQM4A, seg_pos={0 0}, upload=529831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 30 19:59:26.006597174 (604h30m25.877680249s) [noise=true , address=45.91.116.10:4081 , piece_id=5OH3T6YWTPNWNAJTGNQIG4TPMWFH646OTTCQDUMYAZUHDT3HQM4A, seg_pos={0 0}, upload=529831]: Long tail cancellation due to optimal threshold Dec 30 19:59:26.006651678 (604h30m25.877734713s) [noise=true , address=145.79.227.49:4338 , piece_id=OYMRWST4K5KFB5PV3HZTOTK2JCV7MPZWEZBVZDHVRM7NJZBLT2NA, seg_pos={0 0}, upload=529831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 30 19:59:26.006659393 (604h30m25.877742428s) [noise=true , address=145.79.227.49:4338 , piece_id=OYMRWST4K5KFB5PV3HZTOTK2JCV7MPZWEZBVZDHVRM7NJZBLT2NA, seg_pos={0 0}, upload=529831]: Long tail cancellation due to optimal threshold Dec 30 19:59:26.006668790 (604h30m25.877751826s) [noise=true , address=145.79.227.30:4762 , piece_id=4SRJQBHND2IXLDZSS3ZH7KKYSKBTMNIYEGFF7742NERNJYJYALVQ, seg_pos={0 0}, upload=529831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 30 19:59:26.006673119 (604h30m25.877756154s) [noise=true , address=145.79.227.30:4762 , piece_id=4SRJQBHND2IXLDZSS3ZH7KKYSKBTMNIYEGFF7742NERNJYJYALVQ, seg_pos={0 0}, upload=529831]: Long tail cancellation due to optimal threshold Dec 30 19:59:26.006690051 (604h30m25.877773086s) [seg_pos={0 0}, upload=529831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 19:59:26.006786083 (604h30m25.877869119s) [upload=529831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:59:26.006791164 (604h30m25.877874209s) [upload=529831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:59:26.299587881 (604h30m26.170670916s) [upload=529831]: Done uploading segments... Dec 30 19:59:26.299606246 (604h30m26.170689291s) [upload=529831]: Upload finished. err: <nil> Duration: 6.022580098s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X.....................X......................X....XX.......] duration: 25.316121795s trace: https://jaeger.dev.storj.io/trace/840262455c60921 FAIL (2025-12-30 20:02:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ, Piece ID: MNIZD24L22A677AVYFP2JTJ76YUO73X3YOJ3GBOBTEUTXLCORIEQ) context deadline exceeded; piecestore: (Node ID: 1WpVJa7SXkANKParDWYrC1cN8aLHTkUSPrRJGKQ6fvA5oDEkoa, Piece ID: LLKCRW4FFBCZJYEONAPY2PBBIUA4PHJ6VWGGARB6DEDO7KONFSXA) context deadline exceeded; piecestore: (Node ID: 1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, Piece ID: FD6B7RMU7XCL6ZI4FSG77JBTFGJNORR4CYN4RBRXKH7UGYQVCHUA) context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.40:4075 , piece_id=DZXOXPZ73ZEAWIDRN7AASB4EO4INTJHOVJ342HD5O4CYONEVAICQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.992183721 (604h27m6.863266766s) [noise=true , address=5.231.73.5:4019 , piece_id=DLKN6PTMXQGLSGHSCXJLRCTJ3QKAVXHXTE33R66HPMBMFMNZWQDA, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.992249356 (604h27m6.863332391s) [noise=true , address=77.90.31.10:4521 , piece_id=LLKCRW4FFBCZJYEONAPY2PBBIUA4PHJ6VWGGARB6DEDO7KONFSXA, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.992522767 (604h27m6.863605802s) [noise=true , address=145.79.227.6:4281 , piece_id=IJFEH45LLCFVNT3VURMQ2TWKDTKAPDLIA6DRP4ECGA6M36IHURVQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.992532625 (604h27m6.86361566s) [noise=true , address=145.79.227.39:4016 , piece_id=UK7PLDGLWUXXFBETE2T3LFBHIYHLO6Y4422XYN36AOXHCI3LBFFQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.992817888 (604h27m6.863900933s) [noise=true , address=145.79.227.22:4537 , piece_id=MKOEP4MXR2SJX4EDOIY7UDJ4G25Y5FAJ4AVAZ6KBFYKG45EDJ4TQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.993072103 (604h27m6.864155138s) [noise=true , address=77.90.31.12:4601 , piece_id=Z2TMMXC2Q3O5E4WWZ6PMZGWOBNUPSUMNKUPZCFKJOD6TV2QVDXNQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.993346675 (604h27m6.86442971s) [noise=true , address=145.79.227.49:4344 , piece_id=5DR3HZYF36VOGJ53WOZSB454C7MOWBOYE7U6IGGVTMTG4GO5DA5A, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.993630466 (604h27m6.864713501s) [noise=true , address=77.90.31.21:4891 , piece_id=KPW3XZXEW3BVHIITM4HKYFU5BEF4SVOOYRFWZENES7F7MBD2SX2A, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.993927371 (604h27m6.865010416s) [noise=true , address=145.79.227.30:4775 , piece_id=OYD6XLL2744IECKVTY5ECJUHCEWS4O7LALWZRRCNYBGER4LYVJVQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.994315810 (604h27m6.865398846s) [noise=true , address=77.90.31.17:4790 , piece_id=FD6B7RMU7XCL6ZI4FSG77JBTFGJNORR4CYN4RBRXKH7UGYQVCHUA, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.994706845 (604h27m6.86578988s) [noise=true , address=145.79.227.34:4927 , piece_id=MQZMI72BZ6JXVA3W3YHKSAYMNTPQWEPYX57TETX2QOX4BHWKZ2WA, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.995103571 (604h27m6.866186606s) [noise=true , address=45.91.116.13:4176 , piece_id=MNIZD24L22A677AVYFP2JTJ76YUO73X3YOJ3GBOBTEUTXLCORIEQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.995463917 (604h27m6.866546952s) [noise=true , address=64.49.13.139:4475 , piece_id=7JNXM7PAXURJP2X774PZ56DOFHKXMSUDJGYOXKURIRGZFONRVVOQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.996040635 (604h27m6.86712367s) [noise=true , address=145.79.227.21:4480 , piece_id=L62OLPCCYDK5EDOPPTKEITJOIXYQJZKROEP3YES6EQSHPUR7B2BA, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:08.448203770 (604h27m8.319286805s) [upload=529788]: Next returned nil segment Dec 30 19:56:08.448226784 (604h27m8.319309819s) [upload=529788]: Waiting for error group managing segments... Dec 30 19:56:08.856597951 (604h27m8.727680986s) [noise=true , address=145.79.227.22:4537 , piece_id=MKOEP4MXR2SJX4EDOIY7UDJ4G25Y5FAJ4AVAZ6KBFYKG45EDJ4TQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:08.972642035 (604h27m8.84372508s) [noise=true , address=64.49.13.139:4475 , piece_id=7JNXM7PAXURJP2X774PZ56DOFHKXMSUDJGYOXKURIRGZFONRVVOQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:08.982556104 (604h27m8.853639149s) [noise=true , address=5.231.73.5:4019 , piece_id=DLKN6PTMXQGLSGHSCXJLRCTJ3QKAVXHXTE33R66HPMBMFMNZWQDA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.017622445 (604h27m8.88870549s) [noise=true , address=77.90.31.10:4521 , piece_id=LLKCRW4FFBCZJYEONAPY2PBBIUA4PHJ6VWGGARB6DEDO7KONFSXA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.096193105 (604h27m8.96727614s) [noise=true , address=77.90.31.12:4601 , piece_id=Z2TMMXC2Q3O5E4WWZ6PMZGWOBNUPSUMNKUPZCFKJOD6TV2QVDXNQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.120549052 (604h27m8.991632097s) [noise=true , address=77.90.31.13:4655 , piece_id=ALKFNT4I4ZBK24CFWVVFHJNH4FKT6X4QO5LU5IQXLDSBNGU5FY6Q, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.121572410 (604h27m8.992655455s) [noise=true , address=77.90.31.21:4891 , piece_id=KPW3XZXEW3BVHIITM4HKYFU5BEF4SVOOYRFWZENES7F7MBD2SX2A, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.171218263 (604h27m9.042301298s) [noise=true , address=45.91.116.13:4176 , piece_id=MNIZD24L22A677AVYFP2JTJ76YUO73X3YOJ3GBOBTEUTXLCORIEQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.480613004 (604h27m9.351696049s) [noise=true , address=77.90.31.17:4790 , piece_id=FD6B7RMU7XCL6ZI4FSG77JBTFGJNORR4CYN4RBRXKH7UGYQVCHUA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.779927943 (604h27m9.651010978s) [noise=true , address=145.79.227.30:4775 , piece_id=OYD6XLL2744IECKVTY5ECJUHCEWS4O7LALWZRRCNYBGER4LYVJVQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.809206594 (604h27m9.680289649s) [noise=true , address=145.79.227.49:4344 , piece_id=5DR3HZYF36VOGJ53WOZSB454C7MOWBOYE7U6IGGVTMTG4GO5DA5A, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.882970882 (604h27m9.754053917s) [noise=true , address=145.79.227.34:4927 , piece_id=MQZMI72BZ6JXVA3W3YHKSAYMNTPQWEPYX57TETX2QOX4BHWKZ2WA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.894950363 (604h27m9.766033398s) [noise=true , address=145.79.227.44:4198 , piece_id=HTTP7BJLLNNP7BKCKC43VRRTJWESVCNS6HDMGEPLRN5GMVQHHQKA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.895290701 (604h27m9.766373736s) [noise=true , address=145.79.227.43:4189 , piece_id=SAUQY62J6PECMGHJQTSXRD6NNFWFE6A5AHVQYA3UW3DHLZUXFPIQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.895298405 (604h27m9.766381451s) [seg_pos={0 0}, upload=529788]: Segment reached optimal threshold of 14 pieces. Dec 30 19:56:09.895436178 (604h27m9.766519213s) [noise=true , address=145.79.227.21:4480 , piece_id=L62OLPCCYDK5EDOPPTKEITJOIXYQJZKROEP3YES6EQSHPUR7B2BA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qAtwKs88zrSjvRR9fvez9J9HnRBF2Ax7APfCq9BLHSFTE24a6): protocol: write tcp 188.166.225.170:59714->145.79.227.21:4480: use of closed network connection; write tcp 188.166.225.170:59714->145.79.227.21:4480: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59714->145.79.227.21:4480: use of closed network connection Dec 30 19:56:09.895442039 (604h27m9.766525074s) [noise=true , address=145.79.227.21:4480 , piece_id=L62OLPCCYDK5EDOPPTKEITJOIXYQJZKROEP3YES6EQSHPUR7B2BA, seg_pos={0 0}, upload=529788]: Long tail cancellation due to optimal threshold Dec 30 19:56:09.895651538 (604h27m9.766734573s) [noise=true , address=145.79.227.39:4016 , piece_id=UK7PLDGLWUXXFBETE2T3LFBHIYHLO6Y4422XYN36AOXHCI3LBFFQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jC98jEWn6K8jFHPA9knJeHAkuNEmcS7ThxUvbySDfbq6FmKVR): protocol: write tcp 188.166.225.170:39076->145.79.227.39:4016: use of closed network connection; write tcp 188.166.225.170:39076->145.79.227.39:4016: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39076->145.79.227.39:4016: use of closed network connection Dec 30 19:56:09.895656938 (604h27m9.766739973s) [noise=true , address=145.79.227.39:4016 , piece_id=UK7PLDGLWUXXFBETE2T3LFBHIYHLO6Y4422XYN36AOXHCI3LBFFQ, seg_pos={0 0}, upload=529788]: Long tail cancellation due to optimal threshold Dec 30 19:56:09.895753382 (604h27m9.766836417s) [noise=true , address=145.79.227.40:4075 , piece_id=DZXOXPZ73ZEAWIDRN7AASB4EO4INTJHOVJ342HD5O4CYONEVAICQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 19:56:09.895758010 (604h27m9.766841045s) [noise=true , address=145.79.227.40:4075 , piece_id=DZXOXPZ73ZEAWIDRN7AASB4EO4INTJHOVJ342HD5O4CYONEVAICQ, seg_pos={0 0}, upload=529788]: Long tail cancellation due to optimal threshold Dec 30 19:56:09.895833083 (604h27m9.766916118s) [noise=true , address=145.79.227.6:4281 , piece_id=IJFEH45LLCFVNT3VURMQ2TWKDTKAPDLIA6DRP4ECGA6M36IHURVQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hDXeAm4tomp1gAkBNjM8x77V76yiruYQmPQ3BZEYPLpJzigo2): protocol: expected piece hash; context canceled; EOF Dec 30 19:56:09.895837542 (604h27m9.766920577s) [noise=true , address=145.79.227.6:4281 , piece_id=IJFEH45LLCFVNT3VURMQ2TWKDTKAPDLIA6DRP4ECGA6M36IHURVQ, seg_pos={0 0}, upload=529788]: Long tail cancellation due to optimal threshold Dec 30 19:56:09.895850146 (604h27m9.766933191s) [seg_pos={0 0}, upload=529788]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 19:56:09.895926181 (604h27m9.767009216s) [upload=529788]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:56:09.895933535 (604h27m9.76701657s) [upload=529788]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:56:10.230726588 (604h27m10.101809633s) [upload=529788]: Done uploading segments... Dec 30 19:56:10.230750393 (604h27m10.101833439s) [upload=529788]: Upload finished. err: <nil> Duration: 25.316153837s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X.....................X......................X....XX......] duration: 21.521278191s trace: https://jaeger.dev.storj.io/trace/840262455c60921 FAIL (2025-12-30 20:01:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ, Piece ID: MNIZD24L22A677AVYFP2JTJ76YUO73X3YOJ3GBOBTEUTXLCORIEQ) context deadline exceeded; piecestore: (Node ID: 1WpVJa7SXkANKParDWYrC1cN8aLHTkUSPrRJGKQ6fvA5oDEkoa, Piece ID: LLKCRW4FFBCZJYEONAPY2PBBIUA4PHJ6VWGGARB6DEDO7KONFSXA) context deadline exceeded; piecestore: (Node ID: 1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, Piece ID: FD6B7RMU7XCL6ZI4FSG77JBTFGJNORR4CYN4RBRXKH7UGYQVCHUA) context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.40:4075 , piece_id=DZXOXPZ73ZEAWIDRN7AASB4EO4INTJHOVJ342HD5O4CYONEVAICQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.992183721 (604h27m6.863266766s) [noise=true , address=5.231.73.5:4019 , piece_id=DLKN6PTMXQGLSGHSCXJLRCTJ3QKAVXHXTE33R66HPMBMFMNZWQDA, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.992249356 (604h27m6.863332391s) [noise=true , address=77.90.31.10:4521 , piece_id=LLKCRW4FFBCZJYEONAPY2PBBIUA4PHJ6VWGGARB6DEDO7KONFSXA, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.992522767 (604h27m6.863605802s) [noise=true , address=145.79.227.6:4281 , piece_id=IJFEH45LLCFVNT3VURMQ2TWKDTKAPDLIA6DRP4ECGA6M36IHURVQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.992532625 (604h27m6.86361566s) [noise=true , address=145.79.227.39:4016 , piece_id=UK7PLDGLWUXXFBETE2T3LFBHIYHLO6Y4422XYN36AOXHCI3LBFFQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.992817888 (604h27m6.863900933s) [noise=true , address=145.79.227.22:4537 , piece_id=MKOEP4MXR2SJX4EDOIY7UDJ4G25Y5FAJ4AVAZ6KBFYKG45EDJ4TQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.993072103 (604h27m6.864155138s) [noise=true , address=77.90.31.12:4601 , piece_id=Z2TMMXC2Q3O5E4WWZ6PMZGWOBNUPSUMNKUPZCFKJOD6TV2QVDXNQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.993346675 (604h27m6.86442971s) [noise=true , address=145.79.227.49:4344 , piece_id=5DR3HZYF36VOGJ53WOZSB454C7MOWBOYE7U6IGGVTMTG4GO5DA5A, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.993630466 (604h27m6.864713501s) [noise=true , address=77.90.31.21:4891 , piece_id=KPW3XZXEW3BVHIITM4HKYFU5BEF4SVOOYRFWZENES7F7MBD2SX2A, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.993927371 (604h27m6.865010416s) [noise=true , address=145.79.227.30:4775 , piece_id=OYD6XLL2744IECKVTY5ECJUHCEWS4O7LALWZRRCNYBGER4LYVJVQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.994315810 (604h27m6.865398846s) [noise=true , address=77.90.31.17:4790 , piece_id=FD6B7RMU7XCL6ZI4FSG77JBTFGJNORR4CYN4RBRXKH7UGYQVCHUA, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.994706845 (604h27m6.86578988s) [noise=true , address=145.79.227.34:4927 , piece_id=MQZMI72BZ6JXVA3W3YHKSAYMNTPQWEPYX57TETX2QOX4BHWKZ2WA, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.995103571 (604h27m6.866186606s) [noise=true , address=45.91.116.13:4176 , piece_id=MNIZD24L22A677AVYFP2JTJ76YUO73X3YOJ3GBOBTEUTXLCORIEQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.995463917 (604h27m6.866546952s) [noise=true , address=64.49.13.139:4475 , piece_id=7JNXM7PAXURJP2X774PZ56DOFHKXMSUDJGYOXKURIRGZFONRVVOQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.996040635 (604h27m6.86712367s) [noise=true , address=145.79.227.21:4480 , piece_id=L62OLPCCYDK5EDOPPTKEITJOIXYQJZKROEP3YES6EQSHPUR7B2BA, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:08.448203770 (604h27m8.319286805s) [upload=529788]: Next returned nil segment Dec 30 19:56:08.448226784 (604h27m8.319309819s) [upload=529788]: Waiting for error group managing segments... Dec 30 19:56:08.856597951 (604h27m8.727680986s) [noise=true , address=145.79.227.22:4537 , piece_id=MKOEP4MXR2SJX4EDOIY7UDJ4G25Y5FAJ4AVAZ6KBFYKG45EDJ4TQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:08.972642035 (604h27m8.84372508s) [noise=true , address=64.49.13.139:4475 , piece_id=7JNXM7PAXURJP2X774PZ56DOFHKXMSUDJGYOXKURIRGZFONRVVOQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:08.982556104 (604h27m8.853639149s) [noise=true , address=5.231.73.5:4019 , piece_id=DLKN6PTMXQGLSGHSCXJLRCTJ3QKAVXHXTE33R66HPMBMFMNZWQDA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.017622445 (604h27m8.88870549s) [noise=true , address=77.90.31.10:4521 , piece_id=LLKCRW4FFBCZJYEONAPY2PBBIUA4PHJ6VWGGARB6DEDO7KONFSXA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.096193105 (604h27m8.96727614s) [noise=true , address=77.90.31.12:4601 , piece_id=Z2TMMXC2Q3O5E4WWZ6PMZGWOBNUPSUMNKUPZCFKJOD6TV2QVDXNQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.120549052 (604h27m8.991632097s) [noise=true , address=77.90.31.13:4655 , piece_id=ALKFNT4I4ZBK24CFWVVFHJNH4FKT6X4QO5LU5IQXLDSBNGU5FY6Q, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.121572410 (604h27m8.992655455s) [noise=true , address=77.90.31.21:4891 , piece_id=KPW3XZXEW3BVHIITM4HKYFU5BEF4SVOOYRFWZENES7F7MBD2SX2A, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.171218263 (604h27m9.042301298s) [noise=true , address=45.91.116.13:4176 , piece_id=MNIZD24L22A677AVYFP2JTJ76YUO73X3YOJ3GBOBTEUTXLCORIEQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.480613004 (604h27m9.351696049s) [noise=true , address=77.90.31.17:4790 , piece_id=FD6B7RMU7XCL6ZI4FSG77JBTFGJNORR4CYN4RBRXKH7UGYQVCHUA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.779927943 (604h27m9.651010978s) [noise=true , address=145.79.227.30:4775 , piece_id=OYD6XLL2744IECKVTY5ECJUHCEWS4O7LALWZRRCNYBGER4LYVJVQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.809206594 (604h27m9.680289649s) [noise=true , address=145.79.227.49:4344 , piece_id=5DR3HZYF36VOGJ53WOZSB454C7MOWBOYE7U6IGGVTMTG4GO5DA5A, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.882970882 (604h27m9.754053917s) [noise=true , address=145.79.227.34:4927 , piece_id=MQZMI72BZ6JXVA3W3YHKSAYMNTPQWEPYX57TETX2QOX4BHWKZ2WA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.894950363 (604h27m9.766033398s) [noise=true , address=145.79.227.44:4198 , piece_id=HTTP7BJLLNNP7BKCKC43VRRTJWESVCNS6HDMGEPLRN5GMVQHHQKA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.895290701 (604h27m9.766373736s) [noise=true , address=145.79.227.43:4189 , piece_id=SAUQY62J6PECMGHJQTSXRD6NNFWFE6A5AHVQYA3UW3DHLZUXFPIQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.895298405 (604h27m9.766381451s) [seg_pos={0 0}, upload=529788]: Segment reached optimal threshold of 14 pieces. Dec 30 19:56:09.895436178 (604h27m9.766519213s) [noise=true , address=145.79.227.21:4480 , piece_id=L62OLPCCYDK5EDOPPTKEITJOIXYQJZKROEP3YES6EQSHPUR7B2BA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qAtwKs88zrSjvRR9fvez9J9HnRBF2Ax7APfCq9BLHSFTE24a6): protocol: write tcp 188.166.225.170:59714->145.79.227.21:4480: use of closed network connection; write tcp 188.166.225.170:59714->145.79.227.21:4480: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59714->145.79.227.21:4480: use of closed network connection Dec 30 19:56:09.895442039 (604h27m9.766525074s) [noise=true , address=145.79.227.21:4480 , piece_id=L62OLPCCYDK5EDOPPTKEITJOIXYQJZKROEP3YES6EQSHPUR7B2BA, seg_pos={0 0}, upload=529788]: Long tail cancellation due to optimal threshold Dec 30 19:56:09.895651538 (604h27m9.766734573s) [noise=true , address=145.79.227.39:4016 , piece_id=UK7PLDGLWUXXFBETE2T3LFBHIYHLO6Y4422XYN36AOXHCI3LBFFQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jC98jEWn6K8jFHPA9knJeHAkuNEmcS7ThxUvbySDfbq6FmKVR): protocol: write tcp 188.166.225.170:39076->145.79.227.39:4016: use of closed network connection; write tcp 188.166.225.170:39076->145.79.227.39:4016: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39076->145.79.227.39:4016: use of closed network connection Dec 30 19:56:09.895656938 (604h27m9.766739973s) [noise=true , address=145.79.227.39:4016 , piece_id=UK7PLDGLWUXXFBETE2T3LFBHIYHLO6Y4422XYN36AOXHCI3LBFFQ, seg_pos={0 0}, upload=529788]: Long tail cancellation due to optimal threshold Dec 30 19:56:09.895753382 (604h27m9.766836417s) [noise=true , address=145.79.227.40:4075 , piece_id=DZXOXPZ73ZEAWIDRN7AASB4EO4INTJHOVJ342HD5O4CYONEVAICQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 19:56:09.895758010 (604h27m9.766841045s) [noise=true , address=145.79.227.40:4075 , piece_id=DZXOXPZ73ZEAWIDRN7AASB4EO4INTJHOVJ342HD5O4CYONEVAICQ, seg_pos={0 0}, upload=529788]: Long tail cancellation due to optimal threshold Dec 30 19:56:09.895833083 (604h27m9.766916118s) [noise=true , address=145.79.227.6:4281 , piece_id=IJFEH45LLCFVNT3VURMQ2TWKDTKAPDLIA6DRP4ECGA6M36IHURVQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hDXeAm4tomp1gAkBNjM8x77V76yiruYQmPQ3BZEYPLpJzigo2): protocol: expected piece hash; context canceled; EOF Dec 30 19:56:09.895837542 (604h27m9.766920577s) [noise=true , address=145.79.227.6:4281 , piece_id=IJFEH45LLCFVNT3VURMQ2TWKDTKAPDLIA6DRP4ECGA6M36IHURVQ, seg_pos={0 0}, upload=529788]: Long tail cancellation due to optimal threshold Dec 30 19:56:09.895850146 (604h27m9.766933191s) [seg_pos={0 0}, upload=529788]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 19:56:09.895926181 (604h27m9.767009216s) [upload=529788]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:56:09.895933535 (604h27m9.76701657s) [upload=529788]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:56:10.230726588 (604h27m10.101809633s) [upload=529788]: Done uploading segments... Dec 30 19:56:10.230750393 (604h27m10.101833439s) [upload=529788]: Upload finished. err: <nil> Duration: 21.521304932s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..X..X.....XX.X.....X.....X.................X...X.X.X...X...] duration: 6.003174546s trace: https://jaeger.dev.storj.io/trace/49e32ea040e6dc54 FAIL (2025-12-30 20:01:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: UAV7EMD24JN2CVJ3WG5C5HDYHBTCW4TYLGC35QZI35A7EKIMDVGA) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VLIHBJ7BBCKC7FC53TNNOWDXLWI5C7FOJ3RPHCXPTZM7JVI3P7PA) context deadline exceeded ======LOGS====== Dec 30 19:59:22.789859524 (604h30m22.660942559s) [upload=529831]: Starting upload Dec 30 19:59:22.789937492 (604h30m22.661020537s) [upload=529831]: Done starting upload Dec 30 19:59:22.790028305 (604h30m22.66111134s) [upload=529831]: Uploading segments... Dec 30 19:59:22.790207306 (604h30m22.661290341s) [upload=529831]: Got next segment. Inline: false Dec 30 19:59:22.790263323 (604h30m22.661346378s) [upload=529831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 19:59:22.790269495 (604h30m22.66135254s) [upload=529831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 19:59:23.044447967 (604h30m22.915531042s) [seg_pos={0 0}, upload=529831]: Begin upload segment... Dec 30 19:59:23.044503883 (604h30m22.915586938s) [seg_pos={0 0}, upload=529831]: Done begin upload segment. Dec 30 19:59:23.044537507 (604h30m22.915620543s) [noise=true , address=145.79.227.38:4978 , piece_id=XAWLFWG3GHGTY2WVIJDMLF3QY27LNYQFMGDLNK3HCJZPQSTBXY3A, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.044594004 (604h30m22.915677039s) [noise=true , address=45.91.116.6:4975 , piece_id=QNI5FNAKGYEOP3HDSI53FU4AQPV6PUQCK4SGBXCQRL3AJCZBUXPA, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.045080781 (604h30m22.916163816s) [noise=true , address=77.90.31.10:4517 , piece_id=5V4O7IOLKG7SMJDY6RCWGDJH222XCXTNEZ5YQO2ET5ZQQTU6LB3Q, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.045471716 (604h30m22.916554751s) [noise=true , address=145.79.227.13:4802 , piece_id=J3PGR67GK33SG2CTVCWN7AZDWUTMZ577ZRFWLSVKB3657CHJUMIQ, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.045918206 (604h30m22.917001251s) [noise=true , address=45.91.116.11:4117 , piece_id=VLIHBJ7BBCKC7FC53TNNOWDXLWI5C7FOJ3RPHCXPTZM7JVI3P7PA, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.046449959 (604h30m22.917533004s) [noise=true , address=145.79.227.31:4797 , piece_id=RKCEF2OC2MXOU3CQURIWIVGEAAJFKIHF43H3UPKICRDIUK4OUH4A, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.046883935 (604h30m22.91796698s) [noise=true , address=145.79.227.49:4338 , piece_id=OYMRWST4K5KFB5PV3HZTOTK2JCV7MPZWEZBVZDHVRM7NJZBLT2NA, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.047737561 (604h30m22.918820606s) [noise=true , address=145.79.227.42:4121 , piece_id=ECWXPU2NM6HHSFFUZJAHMZPIZL6GN3WKFPBGCGOLCLZOFCZPD5KQ, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.049760984 (604h30m22.920844029s) [noise=true , address=145.79.227.12:4765 , piece_id=FCRGW3R3T2H5OOCTXGAUL5WWEX5C7O4LQR5LGYSRMAGDD7CIQW4Q, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.049887454 (604h30m22.920970499s) [noise=true , address=45.91.116.10:4081 , piece_id=5OH3T6YWTPNWNAJTGNQIG4TPMWFH646OTTCQDUMYAZUHDT3HQM4A, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.053052571 (604h30m22.924135616s) [noise=true , address=145.79.227.11:4731 , piece_id=XUZBIIS3CB73NCCLURSX2Z4DUCQ6L2EWCE53I4XB62KOUSDYWAIA, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.055404520 (604h30m22.926487565s) [noise=true , address=145.79.227.45:4229 , piece_id=BB6HAB7WGPNLXIZCQTNDHNNXMG2VGHCMGVPSPV7DDYMQIFUC4GLA, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.059215657 (604h30m22.930298692s) [noise=true , address=77.90.31.21:4878 , piece_id=CALPIEYPGBYEVEGBEMM4TVCBSAT3ELHL3S7EDMW3EMSEDKOCBVXA, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.063539760 (604h30m22.934622805s) [noise=true , address=145.79.227.30:4762 , piece_id=4SRJQBHND2IXLDZSS3ZH7KKYSKBTMNIYEGFF7742NERNJYJYALVQ, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.064363259 (604h30m22.935446294s) [noise=true , address=145.79.227.4:4185 , piece_id=OQIBCFJYWU5J7VHH2GXOPX3PVLLIRMNFZSCUMYNOLKLTYKGY5QIQ, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.065657292 (604h30m22.936740338s) [noise=true , address=45.91.116.8:4048 , piece_id=UAV7EMD24JN2CVJ3WG5C5HDYHBTCW4TYLGC35QZI35A7EKIMDVGA, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:24.746124353 (604h30m24.617207388s) [upload=529831]: Next returned nil segment Dec 30 19:59:24.746141626 (604h30m24.617224661s) [upload=529831]: Waiting for error group managing segments... Dec 30 19:59:25.024653914 (604h30m24.895736949s) [noise=true , address=145.79.227.13:4802 , piece_id=J3PGR67GK33SG2CTVCWN7AZDWUTMZ577ZRFWLSVKB3657CHJUMIQ, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.035276661 (604h30m24.906359696s) [noise=true , address=145.79.227.38:4978 , piece_id=XAWLFWG3GHGTY2WVIJDMLF3QY27LNYQFMGDLNK3HCJZPQSTBXY3A, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.043199637 (604h30m24.914282672s) [noise=true , address=145.79.227.4:4185 , piece_id=OQIBCFJYWU5J7VHH2GXOPX3PVLLIRMNFZSCUMYNOLKLTYKGY5QIQ, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.054826626 (604h30m24.925909661s) [noise=true , address=145.79.227.42:4121 , piece_id=ECWXPU2NM6HHSFFUZJAHMZPIZL6GN3WKFPBGCGOLCLZOFCZPD5KQ, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.064076520 (604h30m24.935159565s) [noise=true , address=145.79.227.12:4765 , piece_id=FCRGW3R3T2H5OOCTXGAUL5WWEX5C7O4LQR5LGYSRMAGDD7CIQW4Q, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.067094476 (604h30m24.938177511s) [noise=true , address=145.79.227.31:4797 , piece_id=RKCEF2OC2MXOU3CQURIWIVGEAAJFKIHF43H3UPKICRDIUK4OUH4A, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.134512319 (604h30m25.005595354s) [noise=true , address=145.79.227.11:4731 , piece_id=XUZBIIS3CB73NCCLURSX2Z4DUCQ6L2EWCE53I4XB62KOUSDYWAIA, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.137865084 (604h30m25.008948119s) [noise=true , address=145.79.227.45:4229 , piece_id=BB6HAB7WGPNLXIZCQTNDHNNXMG2VGHCMGVPSPV7DDYMQIFUC4GLA, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.206505284 (604h30m25.077588329s) [noise=true , address=77.90.31.10:4517 , piece_id=5V4O7IOLKG7SMJDY6RCWGDJH222XCXTNEZ5YQO2ET5ZQQTU6LB3Q, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.347801863 (604h30m25.218884898s) [noise=true , address=45.91.116.8:4048 , piece_id=UAV7EMD24JN2CVJ3WG5C5HDYHBTCW4TYLGC35QZI35A7EKIMDVGA, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.373575631 (604h30m25.244658666s) [noise=true , address=45.91.116.6:4975 , piece_id=QNI5FNAKGYEOP3HDSI53FU4AQPV6PUQCK4SGBXCQRL3AJCZBUXPA, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.438579707 (604h30m25.309662752s) [noise=true , address=45.91.116.11:4117 , piece_id=VLIHBJ7BBCKC7FC53TNNOWDXLWI5C7FOJ3RPHCXPTZM7JVI3P7PA, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:26.006186322 (604h30m25.877269367s) [noise=true , address=77.90.31.21:4878 , piece_id=CALPIEYPGBYEVEGBEMM4TVCBSAT3ELHL3S7EDMW3EMSEDKOCBVXA, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:26.006212882 (604h30m25.877295918s) [seg_pos={0 0}, upload=529831]: Segment reached optimal threshold of 13 pieces. Dec 30 19:59:26.006585312 (604h30m25.877668387s) [noise=true , address=45.91.116.10:4081 , piece_id=5OH3T6YWTPNWNAJTGNQIG4TPMWFH646OTTCQDUMYAZUHDT3HQM4A, seg_pos={0 0}, upload=529831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 30 19:59:26.006597174 (604h30m25.877680249s) [noise=true , address=45.91.116.10:4081 , piece_id=5OH3T6YWTPNWNAJTGNQIG4TPMWFH646OTTCQDUMYAZUHDT3HQM4A, seg_pos={0 0}, upload=529831]: Long tail cancellation due to optimal threshold Dec 30 19:59:26.006651678 (604h30m25.877734713s) [noise=true , address=145.79.227.49:4338 , piece_id=OYMRWST4K5KFB5PV3HZTOTK2JCV7MPZWEZBVZDHVRM7NJZBLT2NA, seg_pos={0 0}, upload=529831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 30 19:59:26.006659393 (604h30m25.877742428s) [noise=true , address=145.79.227.49:4338 , piece_id=OYMRWST4K5KFB5PV3HZTOTK2JCV7MPZWEZBVZDHVRM7NJZBLT2NA, seg_pos={0 0}, upload=529831]: Long tail cancellation due to optimal threshold Dec 30 19:59:26.006668790 (604h30m25.877751826s) [noise=true , address=145.79.227.30:4762 , piece_id=4SRJQBHND2IXLDZSS3ZH7KKYSKBTMNIYEGFF7742NERNJYJYALVQ, seg_pos={0 0}, upload=529831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 30 19:59:26.006673119 (604h30m25.877756154s) [noise=true , address=145.79.227.30:4762 , piece_id=4SRJQBHND2IXLDZSS3ZH7KKYSKBTMNIYEGFF7742NERNJYJYALVQ, seg_pos={0 0}, upload=529831]: Long tail cancellation due to optimal threshold Dec 30 19:59:26.006690051 (604h30m25.877773086s) [seg_pos={0 0}, upload=529831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 19:59:26.006786083 (604h30m25.877869119s) [upload=529831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:59:26.006791164 (604h30m25.877874209s) [upload=529831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:59:26.299587881 (604h30m26.170670916s) [upload=529831]: Done uploading segments... Dec 30 19:59:26.299606246 (604h30m26.170689291s) [upload=529831]: Upload finished. err: <nil> Duration: 6.003199864s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [X..X.....................X......................X....XX.....] duration: 15.163781059s trace: https://jaeger.dev.storj.io/trace/840262455c60921 FAIL (2025-12-30 20:01:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ, Piece ID: MNIZD24L22A677AVYFP2JTJ76YUO73X3YOJ3GBOBTEUTXLCORIEQ) context deadline exceeded; piecestore: (Node ID: 1WpVJa7SXkANKParDWYrC1cN8aLHTkUSPrRJGKQ6fvA5oDEkoa, Piece ID: LLKCRW4FFBCZJYEONAPY2PBBIUA4PHJ6VWGGARB6DEDO7KONFSXA) context deadline exceeded; piecestore: (Node ID: 1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, Piece ID: FD6B7RMU7XCL6ZI4FSG77JBTFGJNORR4CYN4RBRXKH7UGYQVCHUA) context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.40:4075 , piece_id=DZXOXPZ73ZEAWIDRN7AASB4EO4INTJHOVJ342HD5O4CYONEVAICQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.992183721 (604h27m6.863266766s) [noise=true , address=5.231.73.5:4019 , piece_id=DLKN6PTMXQGLSGHSCXJLRCTJ3QKAVXHXTE33R66HPMBMFMNZWQDA, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.992249356 (604h27m6.863332391s) [noise=true , address=77.90.31.10:4521 , piece_id=LLKCRW4FFBCZJYEONAPY2PBBIUA4PHJ6VWGGARB6DEDO7KONFSXA, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.992522767 (604h27m6.863605802s) [noise=true , address=145.79.227.6:4281 , piece_id=IJFEH45LLCFVNT3VURMQ2TWKDTKAPDLIA6DRP4ECGA6M36IHURVQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.992532625 (604h27m6.86361566s) [noise=true , address=145.79.227.39:4016 , piece_id=UK7PLDGLWUXXFBETE2T3LFBHIYHLO6Y4422XYN36AOXHCI3LBFFQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.992817888 (604h27m6.863900933s) [noise=true , address=145.79.227.22:4537 , piece_id=MKOEP4MXR2SJX4EDOIY7UDJ4G25Y5FAJ4AVAZ6KBFYKG45EDJ4TQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.993072103 (604h27m6.864155138s) [noise=true , address=77.90.31.12:4601 , piece_id=Z2TMMXC2Q3O5E4WWZ6PMZGWOBNUPSUMNKUPZCFKJOD6TV2QVDXNQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.993346675 (604h27m6.86442971s) [noise=true , address=145.79.227.49:4344 , piece_id=5DR3HZYF36VOGJ53WOZSB454C7MOWBOYE7U6IGGVTMTG4GO5DA5A, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.993630466 (604h27m6.864713501s) [noise=true , address=77.90.31.21:4891 , piece_id=KPW3XZXEW3BVHIITM4HKYFU5BEF4SVOOYRFWZENES7F7MBD2SX2A, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.993927371 (604h27m6.865010416s) [noise=true , address=145.79.227.30:4775 , piece_id=OYD6XLL2744IECKVTY5ECJUHCEWS4O7LALWZRRCNYBGER4LYVJVQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.994315810 (604h27m6.865398846s) [noise=true , address=77.90.31.17:4790 , piece_id=FD6B7RMU7XCL6ZI4FSG77JBTFGJNORR4CYN4RBRXKH7UGYQVCHUA, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.994706845 (604h27m6.86578988s) [noise=true , address=145.79.227.34:4927 , piece_id=MQZMI72BZ6JXVA3W3YHKSAYMNTPQWEPYX57TETX2QOX4BHWKZ2WA, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.995103571 (604h27m6.866186606s) [noise=true , address=45.91.116.13:4176 , piece_id=MNIZD24L22A677AVYFP2JTJ76YUO73X3YOJ3GBOBTEUTXLCORIEQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.995463917 (604h27m6.866546952s) [noise=true , address=64.49.13.139:4475 , piece_id=7JNXM7PAXURJP2X774PZ56DOFHKXMSUDJGYOXKURIRGZFONRVVOQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.996040635 (604h27m6.86712367s) [noise=true , address=145.79.227.21:4480 , piece_id=L62OLPCCYDK5EDOPPTKEITJOIXYQJZKROEP3YES6EQSHPUR7B2BA, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:08.448203770 (604h27m8.319286805s) [upload=529788]: Next returned nil segment Dec 30 19:56:08.448226784 (604h27m8.319309819s) [upload=529788]: Waiting for error group managing segments... Dec 30 19:56:08.856597951 (604h27m8.727680986s) [noise=true , address=145.79.227.22:4537 , piece_id=MKOEP4MXR2SJX4EDOIY7UDJ4G25Y5FAJ4AVAZ6KBFYKG45EDJ4TQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:08.972642035 (604h27m8.84372508s) [noise=true , address=64.49.13.139:4475 , piece_id=7JNXM7PAXURJP2X774PZ56DOFHKXMSUDJGYOXKURIRGZFONRVVOQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:08.982556104 (604h27m8.853639149s) [noise=true , address=5.231.73.5:4019 , piece_id=DLKN6PTMXQGLSGHSCXJLRCTJ3QKAVXHXTE33R66HPMBMFMNZWQDA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.017622445 (604h27m8.88870549s) [noise=true , address=77.90.31.10:4521 , piece_id=LLKCRW4FFBCZJYEONAPY2PBBIUA4PHJ6VWGGARB6DEDO7KONFSXA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.096193105 (604h27m8.96727614s) [noise=true , address=77.90.31.12:4601 , piece_id=Z2TMMXC2Q3O5E4WWZ6PMZGWOBNUPSUMNKUPZCFKJOD6TV2QVDXNQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.120549052 (604h27m8.991632097s) [noise=true , address=77.90.31.13:4655 , piece_id=ALKFNT4I4ZBK24CFWVVFHJNH4FKT6X4QO5LU5IQXLDSBNGU5FY6Q, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.121572410 (604h27m8.992655455s) [noise=true , address=77.90.31.21:4891 , piece_id=KPW3XZXEW3BVHIITM4HKYFU5BEF4SVOOYRFWZENES7F7MBD2SX2A, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.171218263 (604h27m9.042301298s) [noise=true , address=45.91.116.13:4176 , piece_id=MNIZD24L22A677AVYFP2JTJ76YUO73X3YOJ3GBOBTEUTXLCORIEQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.480613004 (604h27m9.351696049s) [noise=true , address=77.90.31.17:4790 , piece_id=FD6B7RMU7XCL6ZI4FSG77JBTFGJNORR4CYN4RBRXKH7UGYQVCHUA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.779927943 (604h27m9.651010978s) [noise=true , address=145.79.227.30:4775 , piece_id=OYD6XLL2744IECKVTY5ECJUHCEWS4O7LALWZRRCNYBGER4LYVJVQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.809206594 (604h27m9.680289649s) [noise=true , address=145.79.227.49:4344 , piece_id=5DR3HZYF36VOGJ53WOZSB454C7MOWBOYE7U6IGGVTMTG4GO5DA5A, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.882970882 (604h27m9.754053917s) [noise=true , address=145.79.227.34:4927 , piece_id=MQZMI72BZ6JXVA3W3YHKSAYMNTPQWEPYX57TETX2QOX4BHWKZ2WA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.894950363 (604h27m9.766033398s) [noise=true , address=145.79.227.44:4198 , piece_id=HTTP7BJLLNNP7BKCKC43VRRTJWESVCNS6HDMGEPLRN5GMVQHHQKA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.895290701 (604h27m9.766373736s) [noise=true , address=145.79.227.43:4189 , piece_id=SAUQY62J6PECMGHJQTSXRD6NNFWFE6A5AHVQYA3UW3DHLZUXFPIQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.895298405 (604h27m9.766381451s) [seg_pos={0 0}, upload=529788]: Segment reached optimal threshold of 14 pieces. Dec 30 19:56:09.895436178 (604h27m9.766519213s) [noise=true , address=145.79.227.21:4480 , piece_id=L62OLPCCYDK5EDOPPTKEITJOIXYQJZKROEP3YES6EQSHPUR7B2BA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qAtwKs88zrSjvRR9fvez9J9HnRBF2Ax7APfCq9BLHSFTE24a6): protocol: write tcp 188.166.225.170:59714->145.79.227.21:4480: use of closed network connection; write tcp 188.166.225.170:59714->145.79.227.21:4480: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59714->145.79.227.21:4480: use of closed network connection Dec 30 19:56:09.895442039 (604h27m9.766525074s) [noise=true , address=145.79.227.21:4480 , piece_id=L62OLPCCYDK5EDOPPTKEITJOIXYQJZKROEP3YES6EQSHPUR7B2BA, seg_pos={0 0}, upload=529788]: Long tail cancellation due to optimal threshold Dec 30 19:56:09.895651538 (604h27m9.766734573s) [noise=true , address=145.79.227.39:4016 , piece_id=UK7PLDGLWUXXFBETE2T3LFBHIYHLO6Y4422XYN36AOXHCI3LBFFQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jC98jEWn6K8jFHPA9knJeHAkuNEmcS7ThxUvbySDfbq6FmKVR): protocol: write tcp 188.166.225.170:39076->145.79.227.39:4016: use of closed network connection; write tcp 188.166.225.170:39076->145.79.227.39:4016: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39076->145.79.227.39:4016: use of closed network connection Dec 30 19:56:09.895656938 (604h27m9.766739973s) [noise=true , address=145.79.227.39:4016 , piece_id=UK7PLDGLWUXXFBETE2T3LFBHIYHLO6Y4422XYN36AOXHCI3LBFFQ, seg_pos={0 0}, upload=529788]: Long tail cancellation due to optimal threshold Dec 30 19:56:09.895753382 (604h27m9.766836417s) [noise=true , address=145.79.227.40:4075 , piece_id=DZXOXPZ73ZEAWIDRN7AASB4EO4INTJHOVJ342HD5O4CYONEVAICQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 19:56:09.895758010 (604h27m9.766841045s) [noise=true , address=145.79.227.40:4075 , piece_id=DZXOXPZ73ZEAWIDRN7AASB4EO4INTJHOVJ342HD5O4CYONEVAICQ, seg_pos={0 0}, upload=529788]: Long tail cancellation due to optimal threshold Dec 30 19:56:09.895833083 (604h27m9.766916118s) [noise=true , address=145.79.227.6:4281 , piece_id=IJFEH45LLCFVNT3VURMQ2TWKDTKAPDLIA6DRP4ECGA6M36IHURVQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hDXeAm4tomp1gAkBNjM8x77V76yiruYQmPQ3BZEYPLpJzigo2): protocol: expected piece hash; context canceled; EOF Dec 30 19:56:09.895837542 (604h27m9.766920577s) [noise=true , address=145.79.227.6:4281 , piece_id=IJFEH45LLCFVNT3VURMQ2TWKDTKAPDLIA6DRP4ECGA6M36IHURVQ, seg_pos={0 0}, upload=529788]: Long tail cancellation due to optimal threshold Dec 30 19:56:09.895850146 (604h27m9.766933191s) [seg_pos={0 0}, upload=529788]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 19:56:09.895926181 (604h27m9.767009216s) [upload=529788]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:56:09.895933535 (604h27m9.76701657s) [upload=529788]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:56:10.230726588 (604h27m10.101809633s) [upload=529788]: Done uploading segments... Dec 30 19:56:10.230750393 (604h27m10.101833439s) [upload=529788]: Upload finished. err: <nil> Duration: 15.163814302s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...X..X.....XX.X.....X.....X.................X...X.X.X...X..] duration: 23.391191538s trace: https://jaeger.dev.storj.io/trace/49e32ea040e6dc54 FAIL (2025-12-30 20:00:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: UAV7EMD24JN2CVJ3WG5C5HDYHBTCW4TYLGC35QZI35A7EKIMDVGA) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VLIHBJ7BBCKC7FC53TNNOWDXLWI5C7FOJ3RPHCXPTZM7JVI3P7PA) context deadline exceeded ======LOGS====== Dec 30 19:59:22.789859524 (604h30m22.660942559s) [upload=529831]: Starting upload Dec 30 19:59:22.789937492 (604h30m22.661020537s) [upload=529831]: Done starting upload Dec 30 19:59:22.790028305 (604h30m22.66111134s) [upload=529831]: Uploading segments... Dec 30 19:59:22.790207306 (604h30m22.661290341s) [upload=529831]: Got next segment. Inline: false Dec 30 19:59:22.790263323 (604h30m22.661346378s) [upload=529831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 19:59:22.790269495 (604h30m22.66135254s) [upload=529831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 19:59:23.044447967 (604h30m22.915531042s) [seg_pos={0 0}, upload=529831]: Begin upload segment... Dec 30 19:59:23.044503883 (604h30m22.915586938s) [seg_pos={0 0}, upload=529831]: Done begin upload segment. Dec 30 19:59:23.044537507 (604h30m22.915620543s) [noise=true , address=145.79.227.38:4978 , piece_id=XAWLFWG3GHGTY2WVIJDMLF3QY27LNYQFMGDLNK3HCJZPQSTBXY3A, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.044594004 (604h30m22.915677039s) [noise=true , address=45.91.116.6:4975 , piece_id=QNI5FNAKGYEOP3HDSI53FU4AQPV6PUQCK4SGBXCQRL3AJCZBUXPA, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.045080781 (604h30m22.916163816s) [noise=true , address=77.90.31.10:4517 , piece_id=5V4O7IOLKG7SMJDY6RCWGDJH222XCXTNEZ5YQO2ET5ZQQTU6LB3Q, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.045471716 (604h30m22.916554751s) [noise=true , address=145.79.227.13:4802 , piece_id=J3PGR67GK33SG2CTVCWN7AZDWUTMZ577ZRFWLSVKB3657CHJUMIQ, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.045918206 (604h30m22.917001251s) [noise=true , address=45.91.116.11:4117 , piece_id=VLIHBJ7BBCKC7FC53TNNOWDXLWI5C7FOJ3RPHCXPTZM7JVI3P7PA, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.046449959 (604h30m22.917533004s) [noise=true , address=145.79.227.31:4797 , piece_id=RKCEF2OC2MXOU3CQURIWIVGEAAJFKIHF43H3UPKICRDIUK4OUH4A, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.046883935 (604h30m22.91796698s) [noise=true , address=145.79.227.49:4338 , piece_id=OYMRWST4K5KFB5PV3HZTOTK2JCV7MPZWEZBVZDHVRM7NJZBLT2NA, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.047737561 (604h30m22.918820606s) [noise=true , address=145.79.227.42:4121 , piece_id=ECWXPU2NM6HHSFFUZJAHMZPIZL6GN3WKFPBGCGOLCLZOFCZPD5KQ, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.049760984 (604h30m22.920844029s) [noise=true , address=145.79.227.12:4765 , piece_id=FCRGW3R3T2H5OOCTXGAUL5WWEX5C7O4LQR5LGYSRMAGDD7CIQW4Q, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.049887454 (604h30m22.920970499s) [noise=true , address=45.91.116.10:4081 , piece_id=5OH3T6YWTPNWNAJTGNQIG4TPMWFH646OTTCQDUMYAZUHDT3HQM4A, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.053052571 (604h30m22.924135616s) [noise=true , address=145.79.227.11:4731 , piece_id=XUZBIIS3CB73NCCLURSX2Z4DUCQ6L2EWCE53I4XB62KOUSDYWAIA, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.055404520 (604h30m22.926487565s) [noise=true , address=145.79.227.45:4229 , piece_id=BB6HAB7WGPNLXIZCQTNDHNNXMG2VGHCMGVPSPV7DDYMQIFUC4GLA, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.059215657 (604h30m22.930298692s) [noise=true , address=77.90.31.21:4878 , piece_id=CALPIEYPGBYEVEGBEMM4TVCBSAT3ELHL3S7EDMW3EMSEDKOCBVXA, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.063539760 (604h30m22.934622805s) [noise=true , address=145.79.227.30:4762 , piece_id=4SRJQBHND2IXLDZSS3ZH7KKYSKBTMNIYEGFF7742NERNJYJYALVQ, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.064363259 (604h30m22.935446294s) [noise=true , address=145.79.227.4:4185 , piece_id=OQIBCFJYWU5J7VHH2GXOPX3PVLLIRMNFZSCUMYNOLKLTYKGY5QIQ, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.065657292 (604h30m22.936740338s) [noise=true , address=45.91.116.8:4048 , piece_id=UAV7EMD24JN2CVJ3WG5C5HDYHBTCW4TYLGC35QZI35A7EKIMDVGA, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:24.746124353 (604h30m24.617207388s) [upload=529831]: Next returned nil segment Dec 30 19:59:24.746141626 (604h30m24.617224661s) [upload=529831]: Waiting for error group managing segments... Dec 30 19:59:25.024653914 (604h30m24.895736949s) [noise=true , address=145.79.227.13:4802 , piece_id=J3PGR67GK33SG2CTVCWN7AZDWUTMZ577ZRFWLSVKB3657CHJUMIQ, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.035276661 (604h30m24.906359696s) [noise=true , address=145.79.227.38:4978 , piece_id=XAWLFWG3GHGTY2WVIJDMLF3QY27LNYQFMGDLNK3HCJZPQSTBXY3A, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.043199637 (604h30m24.914282672s) [noise=true , address=145.79.227.4:4185 , piece_id=OQIBCFJYWU5J7VHH2GXOPX3PVLLIRMNFZSCUMYNOLKLTYKGY5QIQ, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.054826626 (604h30m24.925909661s) [noise=true , address=145.79.227.42:4121 , piece_id=ECWXPU2NM6HHSFFUZJAHMZPIZL6GN3WKFPBGCGOLCLZOFCZPD5KQ, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.064076520 (604h30m24.935159565s) [noise=true , address=145.79.227.12:4765 , piece_id=FCRGW3R3T2H5OOCTXGAUL5WWEX5C7O4LQR5LGYSRMAGDD7CIQW4Q, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.067094476 (604h30m24.938177511s) [noise=true , address=145.79.227.31:4797 , piece_id=RKCEF2OC2MXOU3CQURIWIVGEAAJFKIHF43H3UPKICRDIUK4OUH4A, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.134512319 (604h30m25.005595354s) [noise=true , address=145.79.227.11:4731 , piece_id=XUZBIIS3CB73NCCLURSX2Z4DUCQ6L2EWCE53I4XB62KOUSDYWAIA, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.137865084 (604h30m25.008948119s) [noise=true , address=145.79.227.45:4229 , piece_id=BB6HAB7WGPNLXIZCQTNDHNNXMG2VGHCMGVPSPV7DDYMQIFUC4GLA, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.206505284 (604h30m25.077588329s) [noise=true , address=77.90.31.10:4517 , piece_id=5V4O7IOLKG7SMJDY6RCWGDJH222XCXTNEZ5YQO2ET5ZQQTU6LB3Q, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.347801863 (604h30m25.218884898s) [noise=true , address=45.91.116.8:4048 , piece_id=UAV7EMD24JN2CVJ3WG5C5HDYHBTCW4TYLGC35QZI35A7EKIMDVGA, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.373575631 (604h30m25.244658666s) [noise=true , address=45.91.116.6:4975 , piece_id=QNI5FNAKGYEOP3HDSI53FU4AQPV6PUQCK4SGBXCQRL3AJCZBUXPA, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.438579707 (604h30m25.309662752s) [noise=true , address=45.91.116.11:4117 , piece_id=VLIHBJ7BBCKC7FC53TNNOWDXLWI5C7FOJ3RPHCXPTZM7JVI3P7PA, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:26.006186322 (604h30m25.877269367s) [noise=true , address=77.90.31.21:4878 , piece_id=CALPIEYPGBYEVEGBEMM4TVCBSAT3ELHL3S7EDMW3EMSEDKOCBVXA, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:26.006212882 (604h30m25.877295918s) [seg_pos={0 0}, upload=529831]: Segment reached optimal threshold of 13 pieces. Dec 30 19:59:26.006585312 (604h30m25.877668387s) [noise=true , address=45.91.116.10:4081 , piece_id=5OH3T6YWTPNWNAJTGNQIG4TPMWFH646OTTCQDUMYAZUHDT3HQM4A, seg_pos={0 0}, upload=529831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 30 19:59:26.006597174 (604h30m25.877680249s) [noise=true , address=45.91.116.10:4081 , piece_id=5OH3T6YWTPNWNAJTGNQIG4TPMWFH646OTTCQDUMYAZUHDT3HQM4A, seg_pos={0 0}, upload=529831]: Long tail cancellation due to optimal threshold Dec 30 19:59:26.006651678 (604h30m25.877734713s) [noise=true , address=145.79.227.49:4338 , piece_id=OYMRWST4K5KFB5PV3HZTOTK2JCV7MPZWEZBVZDHVRM7NJZBLT2NA, seg_pos={0 0}, upload=529831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 30 19:59:26.006659393 (604h30m25.877742428s) [noise=true , address=145.79.227.49:4338 , piece_id=OYMRWST4K5KFB5PV3HZTOTK2JCV7MPZWEZBVZDHVRM7NJZBLT2NA, seg_pos={0 0}, upload=529831]: Long tail cancellation due to optimal threshold Dec 30 19:59:26.006668790 (604h30m25.877751826s) [noise=true , address=145.79.227.30:4762 , piece_id=4SRJQBHND2IXLDZSS3ZH7KKYSKBTMNIYEGFF7742NERNJYJYALVQ, seg_pos={0 0}, upload=529831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 30 19:59:26.006673119 (604h30m25.877756154s) [noise=true , address=145.79.227.30:4762 , piece_id=4SRJQBHND2IXLDZSS3ZH7KKYSKBTMNIYEGFF7742NERNJYJYALVQ, seg_pos={0 0}, upload=529831]: Long tail cancellation due to optimal threshold Dec 30 19:59:26.006690051 (604h30m25.877773086s) [seg_pos={0 0}, upload=529831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 19:59:26.006786083 (604h30m25.877869119s) [upload=529831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:59:26.006791164 (604h30m25.877874209s) [upload=529831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:59:26.299587881 (604h30m26.170670916s) [upload=529831]: Done uploading segments... Dec 30 19:59:26.299606246 (604h30m26.170689291s) [upload=529831]: Upload finished. err: <nil> Duration: 23.391225553s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X..X.....................X......................X....XX....] duration: 15.381028592s trace: https://jaeger.dev.storj.io/trace/840262455c60921 FAIL (2025-12-30 20:00:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ, Piece ID: MNIZD24L22A677AVYFP2JTJ76YUO73X3YOJ3GBOBTEUTXLCORIEQ) context deadline exceeded; piecestore: (Node ID: 1WpVJa7SXkANKParDWYrC1cN8aLHTkUSPrRJGKQ6fvA5oDEkoa, Piece ID: LLKCRW4FFBCZJYEONAPY2PBBIUA4PHJ6VWGGARB6DEDO7KONFSXA) context deadline exceeded; piecestore: (Node ID: 1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, Piece ID: FD6B7RMU7XCL6ZI4FSG77JBTFGJNORR4CYN4RBRXKH7UGYQVCHUA) context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.40:4075 , piece_id=DZXOXPZ73ZEAWIDRN7AASB4EO4INTJHOVJ342HD5O4CYONEVAICQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.992183721 (604h27m6.863266766s) [noise=true , address=5.231.73.5:4019 , piece_id=DLKN6PTMXQGLSGHSCXJLRCTJ3QKAVXHXTE33R66HPMBMFMNZWQDA, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.992249356 (604h27m6.863332391s) [noise=true , address=77.90.31.10:4521 , piece_id=LLKCRW4FFBCZJYEONAPY2PBBIUA4PHJ6VWGGARB6DEDO7KONFSXA, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.992522767 (604h27m6.863605802s) [noise=true , address=145.79.227.6:4281 , piece_id=IJFEH45LLCFVNT3VURMQ2TWKDTKAPDLIA6DRP4ECGA6M36IHURVQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.992532625 (604h27m6.86361566s) [noise=true , address=145.79.227.39:4016 , piece_id=UK7PLDGLWUXXFBETE2T3LFBHIYHLO6Y4422XYN36AOXHCI3LBFFQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.992817888 (604h27m6.863900933s) [noise=true , address=145.79.227.22:4537 , piece_id=MKOEP4MXR2SJX4EDOIY7UDJ4G25Y5FAJ4AVAZ6KBFYKG45EDJ4TQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.993072103 (604h27m6.864155138s) [noise=true , address=77.90.31.12:4601 , piece_id=Z2TMMXC2Q3O5E4WWZ6PMZGWOBNUPSUMNKUPZCFKJOD6TV2QVDXNQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.993346675 (604h27m6.86442971s) [noise=true , address=145.79.227.49:4344 , piece_id=5DR3HZYF36VOGJ53WOZSB454C7MOWBOYE7U6IGGVTMTG4GO5DA5A, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.993630466 (604h27m6.864713501s) [noise=true , address=77.90.31.21:4891 , piece_id=KPW3XZXEW3BVHIITM4HKYFU5BEF4SVOOYRFWZENES7F7MBD2SX2A, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.993927371 (604h27m6.865010416s) [noise=true , address=145.79.227.30:4775 , piece_id=OYD6XLL2744IECKVTY5ECJUHCEWS4O7LALWZRRCNYBGER4LYVJVQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.994315810 (604h27m6.865398846s) [noise=true , address=77.90.31.17:4790 , piece_id=FD6B7RMU7XCL6ZI4FSG77JBTFGJNORR4CYN4RBRXKH7UGYQVCHUA, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.994706845 (604h27m6.86578988s) [noise=true , address=145.79.227.34:4927 , piece_id=MQZMI72BZ6JXVA3W3YHKSAYMNTPQWEPYX57TETX2QOX4BHWKZ2WA, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.995103571 (604h27m6.866186606s) [noise=true , address=45.91.116.13:4176 , piece_id=MNIZD24L22A677AVYFP2JTJ76YUO73X3YOJ3GBOBTEUTXLCORIEQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.995463917 (604h27m6.866546952s) [noise=true , address=64.49.13.139:4475 , piece_id=7JNXM7PAXURJP2X774PZ56DOFHKXMSUDJGYOXKURIRGZFONRVVOQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.996040635 (604h27m6.86712367s) [noise=true , address=145.79.227.21:4480 , piece_id=L62OLPCCYDK5EDOPPTKEITJOIXYQJZKROEP3YES6EQSHPUR7B2BA, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:08.448203770 (604h27m8.319286805s) [upload=529788]: Next returned nil segment Dec 30 19:56:08.448226784 (604h27m8.319309819s) [upload=529788]: Waiting for error group managing segments... Dec 30 19:56:08.856597951 (604h27m8.727680986s) [noise=true , address=145.79.227.22:4537 , piece_id=MKOEP4MXR2SJX4EDOIY7UDJ4G25Y5FAJ4AVAZ6KBFYKG45EDJ4TQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:08.972642035 (604h27m8.84372508s) [noise=true , address=64.49.13.139:4475 , piece_id=7JNXM7PAXURJP2X774PZ56DOFHKXMSUDJGYOXKURIRGZFONRVVOQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:08.982556104 (604h27m8.853639149s) [noise=true , address=5.231.73.5:4019 , piece_id=DLKN6PTMXQGLSGHSCXJLRCTJ3QKAVXHXTE33R66HPMBMFMNZWQDA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.017622445 (604h27m8.88870549s) [noise=true , address=77.90.31.10:4521 , piece_id=LLKCRW4FFBCZJYEONAPY2PBBIUA4PHJ6VWGGARB6DEDO7KONFSXA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.096193105 (604h27m8.96727614s) [noise=true , address=77.90.31.12:4601 , piece_id=Z2TMMXC2Q3O5E4WWZ6PMZGWOBNUPSUMNKUPZCFKJOD6TV2QVDXNQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.120549052 (604h27m8.991632097s) [noise=true , address=77.90.31.13:4655 , piece_id=ALKFNT4I4ZBK24CFWVVFHJNH4FKT6X4QO5LU5IQXLDSBNGU5FY6Q, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.121572410 (604h27m8.992655455s) [noise=true , address=77.90.31.21:4891 , piece_id=KPW3XZXEW3BVHIITM4HKYFU5BEF4SVOOYRFWZENES7F7MBD2SX2A, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.171218263 (604h27m9.042301298s) [noise=true , address=45.91.116.13:4176 , piece_id=MNIZD24L22A677AVYFP2JTJ76YUO73X3YOJ3GBOBTEUTXLCORIEQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.480613004 (604h27m9.351696049s) [noise=true , address=77.90.31.17:4790 , piece_id=FD6B7RMU7XCL6ZI4FSG77JBTFGJNORR4CYN4RBRXKH7UGYQVCHUA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.779927943 (604h27m9.651010978s) [noise=true , address=145.79.227.30:4775 , piece_id=OYD6XLL2744IECKVTY5ECJUHCEWS4O7LALWZRRCNYBGER4LYVJVQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.809206594 (604h27m9.680289649s) [noise=true , address=145.79.227.49:4344 , piece_id=5DR3HZYF36VOGJ53WOZSB454C7MOWBOYE7U6IGGVTMTG4GO5DA5A, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.882970882 (604h27m9.754053917s) [noise=true , address=145.79.227.34:4927 , piece_id=MQZMI72BZ6JXVA3W3YHKSAYMNTPQWEPYX57TETX2QOX4BHWKZ2WA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.894950363 (604h27m9.766033398s) [noise=true , address=145.79.227.44:4198 , piece_id=HTTP7BJLLNNP7BKCKC43VRRTJWESVCNS6HDMGEPLRN5GMVQHHQKA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.895290701 (604h27m9.766373736s) [noise=true , address=145.79.227.43:4189 , piece_id=SAUQY62J6PECMGHJQTSXRD6NNFWFE6A5AHVQYA3UW3DHLZUXFPIQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.895298405 (604h27m9.766381451s) [seg_pos={0 0}, upload=529788]: Segment reached optimal threshold of 14 pieces. Dec 30 19:56:09.895436178 (604h27m9.766519213s) [noise=true , address=145.79.227.21:4480 , piece_id=L62OLPCCYDK5EDOPPTKEITJOIXYQJZKROEP3YES6EQSHPUR7B2BA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qAtwKs88zrSjvRR9fvez9J9HnRBF2Ax7APfCq9BLHSFTE24a6): protocol: write tcp 188.166.225.170:59714->145.79.227.21:4480: use of closed network connection; write tcp 188.166.225.170:59714->145.79.227.21:4480: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59714->145.79.227.21:4480: use of closed network connection Dec 30 19:56:09.895442039 (604h27m9.766525074s) [noise=true , address=145.79.227.21:4480 , piece_id=L62OLPCCYDK5EDOPPTKEITJOIXYQJZKROEP3YES6EQSHPUR7B2BA, seg_pos={0 0}, upload=529788]: Long tail cancellation due to optimal threshold Dec 30 19:56:09.895651538 (604h27m9.766734573s) [noise=true , address=145.79.227.39:4016 , piece_id=UK7PLDGLWUXXFBETE2T3LFBHIYHLO6Y4422XYN36AOXHCI3LBFFQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jC98jEWn6K8jFHPA9knJeHAkuNEmcS7ThxUvbySDfbq6FmKVR): protocol: write tcp 188.166.225.170:39076->145.79.227.39:4016: use of closed network connection; write tcp 188.166.225.170:39076->145.79.227.39:4016: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39076->145.79.227.39:4016: use of closed network connection Dec 30 19:56:09.895656938 (604h27m9.766739973s) [noise=true , address=145.79.227.39:4016 , piece_id=UK7PLDGLWUXXFBETE2T3LFBHIYHLO6Y4422XYN36AOXHCI3LBFFQ, seg_pos={0 0}, upload=529788]: Long tail cancellation due to optimal threshold Dec 30 19:56:09.895753382 (604h27m9.766836417s) [noise=true , address=145.79.227.40:4075 , piece_id=DZXOXPZ73ZEAWIDRN7AASB4EO4INTJHOVJ342HD5O4CYONEVAICQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 19:56:09.895758010 (604h27m9.766841045s) [noise=true , address=145.79.227.40:4075 , piece_id=DZXOXPZ73ZEAWIDRN7AASB4EO4INTJHOVJ342HD5O4CYONEVAICQ, seg_pos={0 0}, upload=529788]: Long tail cancellation due to optimal threshold Dec 30 19:56:09.895833083 (604h27m9.766916118s) [noise=true , address=145.79.227.6:4281 , piece_id=IJFEH45LLCFVNT3VURMQ2TWKDTKAPDLIA6DRP4ECGA6M36IHURVQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hDXeAm4tomp1gAkBNjM8x77V76yiruYQmPQ3BZEYPLpJzigo2): protocol: expected piece hash; context canceled; EOF Dec 30 19:56:09.895837542 (604h27m9.766920577s) [noise=true , address=145.79.227.6:4281 , piece_id=IJFEH45LLCFVNT3VURMQ2TWKDTKAPDLIA6DRP4ECGA6M36IHURVQ, seg_pos={0 0}, upload=529788]: Long tail cancellation due to optimal threshold Dec 30 19:56:09.895850146 (604h27m9.766933191s) [seg_pos={0 0}, upload=529788]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 19:56:09.895926181 (604h27m9.767009216s) [upload=529788]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:56:09.895933535 (604h27m9.76701657s) [upload=529788]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:56:10.230726588 (604h27m10.101809633s) [upload=529788]: Done uploading segments... Dec 30 19:56:10.230750393 (604h27m10.101833439s) [upload=529788]: Upload finished. err: <nil> Duration: 15.381077335s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X...X..X.....XX.X.....X.....X.................X...X.X.X...X.] duration: 5.987305182s trace: https://jaeger.dev.storj.io/trace/49e32ea040e6dc54 FAIL (2025-12-30 19:59:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: UAV7EMD24JN2CVJ3WG5C5HDYHBTCW4TYLGC35QZI35A7EKIMDVGA) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VLIHBJ7BBCKC7FC53TNNOWDXLWI5C7FOJ3RPHCXPTZM7JVI3P7PA) context deadline exceeded ======LOGS====== Dec 30 19:59:22.789859524 (604h30m22.660942559s) [upload=529831]: Starting upload Dec 30 19:59:22.789937492 (604h30m22.661020537s) [upload=529831]: Done starting upload Dec 30 19:59:22.790028305 (604h30m22.66111134s) [upload=529831]: Uploading segments... Dec 30 19:59:22.790207306 (604h30m22.661290341s) [upload=529831]: Got next segment. Inline: false Dec 30 19:59:22.790263323 (604h30m22.661346378s) [upload=529831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 19:59:22.790269495 (604h30m22.66135254s) [upload=529831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 19:59:23.044447967 (604h30m22.915531042s) [seg_pos={0 0}, upload=529831]: Begin upload segment... Dec 30 19:59:23.044503883 (604h30m22.915586938s) [seg_pos={0 0}, upload=529831]: Done begin upload segment. Dec 30 19:59:23.044537507 (604h30m22.915620543s) [noise=true , address=145.79.227.38:4978 , piece_id=XAWLFWG3GHGTY2WVIJDMLF3QY27LNYQFMGDLNK3HCJZPQSTBXY3A, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.044594004 (604h30m22.915677039s) [noise=true , address=45.91.116.6:4975 , piece_id=QNI5FNAKGYEOP3HDSI53FU4AQPV6PUQCK4SGBXCQRL3AJCZBUXPA, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.045080781 (604h30m22.916163816s) [noise=true , address=77.90.31.10:4517 , piece_id=5V4O7IOLKG7SMJDY6RCWGDJH222XCXTNEZ5YQO2ET5ZQQTU6LB3Q, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.045471716 (604h30m22.916554751s) [noise=true , address=145.79.227.13:4802 , piece_id=J3PGR67GK33SG2CTVCWN7AZDWUTMZ577ZRFWLSVKB3657CHJUMIQ, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.045918206 (604h30m22.917001251s) [noise=true , address=45.91.116.11:4117 , piece_id=VLIHBJ7BBCKC7FC53TNNOWDXLWI5C7FOJ3RPHCXPTZM7JVI3P7PA, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.046449959 (604h30m22.917533004s) [noise=true , address=145.79.227.31:4797 , piece_id=RKCEF2OC2MXOU3CQURIWIVGEAAJFKIHF43H3UPKICRDIUK4OUH4A, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.046883935 (604h30m22.91796698s) [noise=true , address=145.79.227.49:4338 , piece_id=OYMRWST4K5KFB5PV3HZTOTK2JCV7MPZWEZBVZDHVRM7NJZBLT2NA, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.047737561 (604h30m22.918820606s) [noise=true , address=145.79.227.42:4121 , piece_id=ECWXPU2NM6HHSFFUZJAHMZPIZL6GN3WKFPBGCGOLCLZOFCZPD5KQ, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.049760984 (604h30m22.920844029s) [noise=true , address=145.79.227.12:4765 , piece_id=FCRGW3R3T2H5OOCTXGAUL5WWEX5C7O4LQR5LGYSRMAGDD7CIQW4Q, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.049887454 (604h30m22.920970499s) [noise=true , address=45.91.116.10:4081 , piece_id=5OH3T6YWTPNWNAJTGNQIG4TPMWFH646OTTCQDUMYAZUHDT3HQM4A, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.053052571 (604h30m22.924135616s) [noise=true , address=145.79.227.11:4731 , piece_id=XUZBIIS3CB73NCCLURSX2Z4DUCQ6L2EWCE53I4XB62KOUSDYWAIA, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.055404520 (604h30m22.926487565s) [noise=true , address=145.79.227.45:4229 , piece_id=BB6HAB7WGPNLXIZCQTNDHNNXMG2VGHCMGVPSPV7DDYMQIFUC4GLA, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.059215657 (604h30m22.930298692s) [noise=true , address=77.90.31.21:4878 , piece_id=CALPIEYPGBYEVEGBEMM4TVCBSAT3ELHL3S7EDMW3EMSEDKOCBVXA, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.063539760 (604h30m22.934622805s) [noise=true , address=145.79.227.30:4762 , piece_id=4SRJQBHND2IXLDZSS3ZH7KKYSKBTMNIYEGFF7742NERNJYJYALVQ, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.064363259 (604h30m22.935446294s) [noise=true , address=145.79.227.4:4185 , piece_id=OQIBCFJYWU5J7VHH2GXOPX3PVLLIRMNFZSCUMYNOLKLTYKGY5QIQ, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.065657292 (604h30m22.936740338s) [noise=true , address=45.91.116.8:4048 , piece_id=UAV7EMD24JN2CVJ3WG5C5HDYHBTCW4TYLGC35QZI35A7EKIMDVGA, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:24.746124353 (604h30m24.617207388s) [upload=529831]: Next returned nil segment Dec 30 19:59:24.746141626 (604h30m24.617224661s) [upload=529831]: Waiting for error group managing segments... Dec 30 19:59:25.024653914 (604h30m24.895736949s) [noise=true , address=145.79.227.13:4802 , piece_id=J3PGR67GK33SG2CTVCWN7AZDWUTMZ577ZRFWLSVKB3657CHJUMIQ, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.035276661 (604h30m24.906359696s) [noise=true , address=145.79.227.38:4978 , piece_id=XAWLFWG3GHGTY2WVIJDMLF3QY27LNYQFMGDLNK3HCJZPQSTBXY3A, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.043199637 (604h30m24.914282672s) [noise=true , address=145.79.227.4:4185 , piece_id=OQIBCFJYWU5J7VHH2GXOPX3PVLLIRMNFZSCUMYNOLKLTYKGY5QIQ, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.054826626 (604h30m24.925909661s) [noise=true , address=145.79.227.42:4121 , piece_id=ECWXPU2NM6HHSFFUZJAHMZPIZL6GN3WKFPBGCGOLCLZOFCZPD5KQ, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.064076520 (604h30m24.935159565s) [noise=true , address=145.79.227.12:4765 , piece_id=FCRGW3R3T2H5OOCTXGAUL5WWEX5C7O4LQR5LGYSRMAGDD7CIQW4Q, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.067094476 (604h30m24.938177511s) [noise=true , address=145.79.227.31:4797 , piece_id=RKCEF2OC2MXOU3CQURIWIVGEAAJFKIHF43H3UPKICRDIUK4OUH4A, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.134512319 (604h30m25.005595354s) [noise=true , address=145.79.227.11:4731 , piece_id=XUZBIIS3CB73NCCLURSX2Z4DUCQ6L2EWCE53I4XB62KOUSDYWAIA, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.137865084 (604h30m25.008948119s) [noise=true , address=145.79.227.45:4229 , piece_id=BB6HAB7WGPNLXIZCQTNDHNNXMG2VGHCMGVPSPV7DDYMQIFUC4GLA, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.206505284 (604h30m25.077588329s) [noise=true , address=77.90.31.10:4517 , piece_id=5V4O7IOLKG7SMJDY6RCWGDJH222XCXTNEZ5YQO2ET5ZQQTU6LB3Q, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.347801863 (604h30m25.218884898s) [noise=true , address=45.91.116.8:4048 , piece_id=UAV7EMD24JN2CVJ3WG5C5HDYHBTCW4TYLGC35QZI35A7EKIMDVGA, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.373575631 (604h30m25.244658666s) [noise=true , address=45.91.116.6:4975 , piece_id=QNI5FNAKGYEOP3HDSI53FU4AQPV6PUQCK4SGBXCQRL3AJCZBUXPA, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.438579707 (604h30m25.309662752s) [noise=true , address=45.91.116.11:4117 , piece_id=VLIHBJ7BBCKC7FC53TNNOWDXLWI5C7FOJ3RPHCXPTZM7JVI3P7PA, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:26.006186322 (604h30m25.877269367s) [noise=true , address=77.90.31.21:4878 , piece_id=CALPIEYPGBYEVEGBEMM4TVCBSAT3ELHL3S7EDMW3EMSEDKOCBVXA, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:26.006212882 (604h30m25.877295918s) [seg_pos={0 0}, upload=529831]: Segment reached optimal threshold of 13 pieces. Dec 30 19:59:26.006585312 (604h30m25.877668387s) [noise=true , address=45.91.116.10:4081 , piece_id=5OH3T6YWTPNWNAJTGNQIG4TPMWFH646OTTCQDUMYAZUHDT3HQM4A, seg_pos={0 0}, upload=529831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 30 19:59:26.006597174 (604h30m25.877680249s) [noise=true , address=45.91.116.10:4081 , piece_id=5OH3T6YWTPNWNAJTGNQIG4TPMWFH646OTTCQDUMYAZUHDT3HQM4A, seg_pos={0 0}, upload=529831]: Long tail cancellation due to optimal threshold Dec 30 19:59:26.006651678 (604h30m25.877734713s) [noise=true , address=145.79.227.49:4338 , piece_id=OYMRWST4K5KFB5PV3HZTOTK2JCV7MPZWEZBVZDHVRM7NJZBLT2NA, seg_pos={0 0}, upload=529831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 30 19:59:26.006659393 (604h30m25.877742428s) [noise=true , address=145.79.227.49:4338 , piece_id=OYMRWST4K5KFB5PV3HZTOTK2JCV7MPZWEZBVZDHVRM7NJZBLT2NA, seg_pos={0 0}, upload=529831]: Long tail cancellation due to optimal threshold Dec 30 19:59:26.006668790 (604h30m25.877751826s) [noise=true , address=145.79.227.30:4762 , piece_id=4SRJQBHND2IXLDZSS3ZH7KKYSKBTMNIYEGFF7742NERNJYJYALVQ, seg_pos={0 0}, upload=529831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 30 19:59:26.006673119 (604h30m25.877756154s) [noise=true , address=145.79.227.30:4762 , piece_id=4SRJQBHND2IXLDZSS3ZH7KKYSKBTMNIYEGFF7742NERNJYJYALVQ, seg_pos={0 0}, upload=529831]: Long tail cancellation due to optimal threshold Dec 30 19:59:26.006690051 (604h30m25.877773086s) [seg_pos={0 0}, upload=529831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 19:59:26.006786083 (604h30m25.877869119s) [upload=529831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:59:26.006791164 (604h30m25.877874209s) [upload=529831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:59:26.299587881 (604h30m26.170670916s) [upload=529831]: Done uploading segments... Dec 30 19:59:26.299606246 (604h30m26.170689291s) [upload=529831]: Upload finished. err: <nil> Duration: 5.987320821s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [XX...X..X.....XX.X.....X.....X.................X...X.X.X...X] duration: 30.004393465s trace: https://jaeger.dev.storj.io/trace/49e32ea040e6dc54 FAIL (2025-12-30 19:59:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: UAV7EMD24JN2CVJ3WG5C5HDYHBTCW4TYLGC35QZI35A7EKIMDVGA) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: VLIHBJ7BBCKC7FC53TNNOWDXLWI5C7FOJ3RPHCXPTZM7JVI3P7PA) context deadline exceeded ======LOGS====== Dec 30 19:59:22.789859524 (604h30m22.660942559s) [upload=529831]: Starting upload Dec 30 19:59:22.789937492 (604h30m22.661020537s) [upload=529831]: Done starting upload Dec 30 19:59:22.790028305 (604h30m22.66111134s) [upload=529831]: Uploading segments... Dec 30 19:59:22.790207306 (604h30m22.661290341s) [upload=529831]: Got next segment. Inline: false Dec 30 19:59:22.790263323 (604h30m22.661346378s) [upload=529831]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 19:59:22.790269495 (604h30m22.66135254s) [upload=529831]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 19:59:23.044447967 (604h30m22.915531042s) [seg_pos={0 0}, upload=529831]: Begin upload segment... Dec 30 19:59:23.044503883 (604h30m22.915586938s) [seg_pos={0 0}, upload=529831]: Done begin upload segment. Dec 30 19:59:23.044537507 (604h30m22.915620543s) [noise=true , address=145.79.227.38:4978 , piece_id=XAWLFWG3GHGTY2WVIJDMLF3QY27LNYQFMGDLNK3HCJZPQSTBXY3A, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.044594004 (604h30m22.915677039s) [noise=true , address=45.91.116.6:4975 , piece_id=QNI5FNAKGYEOP3HDSI53FU4AQPV6PUQCK4SGBXCQRL3AJCZBUXPA, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.045080781 (604h30m22.916163816s) [noise=true , address=77.90.31.10:4517 , piece_id=5V4O7IOLKG7SMJDY6RCWGDJH222XCXTNEZ5YQO2ET5ZQQTU6LB3Q, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.045471716 (604h30m22.916554751s) [noise=true , address=145.79.227.13:4802 , piece_id=J3PGR67GK33SG2CTVCWN7AZDWUTMZ577ZRFWLSVKB3657CHJUMIQ, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.045918206 (604h30m22.917001251s) [noise=true , address=45.91.116.11:4117 , piece_id=VLIHBJ7BBCKC7FC53TNNOWDXLWI5C7FOJ3RPHCXPTZM7JVI3P7PA, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.046449959 (604h30m22.917533004s) [noise=true , address=145.79.227.31:4797 , piece_id=RKCEF2OC2MXOU3CQURIWIVGEAAJFKIHF43H3UPKICRDIUK4OUH4A, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.046883935 (604h30m22.91796698s) [noise=true , address=145.79.227.49:4338 , piece_id=OYMRWST4K5KFB5PV3HZTOTK2JCV7MPZWEZBVZDHVRM7NJZBLT2NA, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.047737561 (604h30m22.918820606s) [noise=true , address=145.79.227.42:4121 , piece_id=ECWXPU2NM6HHSFFUZJAHMZPIZL6GN3WKFPBGCGOLCLZOFCZPD5KQ, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.049760984 (604h30m22.920844029s) [noise=true , address=145.79.227.12:4765 , piece_id=FCRGW3R3T2H5OOCTXGAUL5WWEX5C7O4LQR5LGYSRMAGDD7CIQW4Q, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.049887454 (604h30m22.920970499s) [noise=true , address=45.91.116.10:4081 , piece_id=5OH3T6YWTPNWNAJTGNQIG4TPMWFH646OTTCQDUMYAZUHDT3HQM4A, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.053052571 (604h30m22.924135616s) [noise=true , address=145.79.227.11:4731 , piece_id=XUZBIIS3CB73NCCLURSX2Z4DUCQ6L2EWCE53I4XB62KOUSDYWAIA, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.055404520 (604h30m22.926487565s) [noise=true , address=145.79.227.45:4229 , piece_id=BB6HAB7WGPNLXIZCQTNDHNNXMG2VGHCMGVPSPV7DDYMQIFUC4GLA, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.059215657 (604h30m22.930298692s) [noise=true , address=77.90.31.21:4878 , piece_id=CALPIEYPGBYEVEGBEMM4TVCBSAT3ELHL3S7EDMW3EMSEDKOCBVXA, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.063539760 (604h30m22.934622805s) [noise=true , address=145.79.227.30:4762 , piece_id=4SRJQBHND2IXLDZSS3ZH7KKYSKBTMNIYEGFF7742NERNJYJYALVQ, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.064363259 (604h30m22.935446294s) [noise=true , address=145.79.227.4:4185 , piece_id=OQIBCFJYWU5J7VHH2GXOPX3PVLLIRMNFZSCUMYNOLKLTYKGY5QIQ, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:23.065657292 (604h30m22.936740338s) [noise=true , address=45.91.116.8:4048 , piece_id=UAV7EMD24JN2CVJ3WG5C5HDYHBTCW4TYLGC35QZI35A7EKIMDVGA, seg_pos={0 0}, upload=529831]: Uploading piece... Dec 30 19:59:24.746124353 (604h30m24.617207388s) [upload=529831]: Next returned nil segment Dec 30 19:59:24.746141626 (604h30m24.617224661s) [upload=529831]: Waiting for error group managing segments... Dec 30 19:59:25.024653914 (604h30m24.895736949s) [noise=true , address=145.79.227.13:4802 , piece_id=J3PGR67GK33SG2CTVCWN7AZDWUTMZ577ZRFWLSVKB3657CHJUMIQ, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.035276661 (604h30m24.906359696s) [noise=true , address=145.79.227.38:4978 , piece_id=XAWLFWG3GHGTY2WVIJDMLF3QY27LNYQFMGDLNK3HCJZPQSTBXY3A, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.043199637 (604h30m24.914282672s) [noise=true , address=145.79.227.4:4185 , piece_id=OQIBCFJYWU5J7VHH2GXOPX3PVLLIRMNFZSCUMYNOLKLTYKGY5QIQ, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.054826626 (604h30m24.925909661s) [noise=true , address=145.79.227.42:4121 , piece_id=ECWXPU2NM6HHSFFUZJAHMZPIZL6GN3WKFPBGCGOLCLZOFCZPD5KQ, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.064076520 (604h30m24.935159565s) [noise=true , address=145.79.227.12:4765 , piece_id=FCRGW3R3T2H5OOCTXGAUL5WWEX5C7O4LQR5LGYSRMAGDD7CIQW4Q, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.067094476 (604h30m24.938177511s) [noise=true , address=145.79.227.31:4797 , piece_id=RKCEF2OC2MXOU3CQURIWIVGEAAJFKIHF43H3UPKICRDIUK4OUH4A, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.134512319 (604h30m25.005595354s) [noise=true , address=145.79.227.11:4731 , piece_id=XUZBIIS3CB73NCCLURSX2Z4DUCQ6L2EWCE53I4XB62KOUSDYWAIA, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.137865084 (604h30m25.008948119s) [noise=true , address=145.79.227.45:4229 , piece_id=BB6HAB7WGPNLXIZCQTNDHNNXMG2VGHCMGVPSPV7DDYMQIFUC4GLA, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.206505284 (604h30m25.077588329s) [noise=true , address=77.90.31.10:4517 , piece_id=5V4O7IOLKG7SMJDY6RCWGDJH222XCXTNEZ5YQO2ET5ZQQTU6LB3Q, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.347801863 (604h30m25.218884898s) [noise=true , address=45.91.116.8:4048 , piece_id=UAV7EMD24JN2CVJ3WG5C5HDYHBTCW4TYLGC35QZI35A7EKIMDVGA, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.373575631 (604h30m25.244658666s) [noise=true , address=45.91.116.6:4975 , piece_id=QNI5FNAKGYEOP3HDSI53FU4AQPV6PUQCK4SGBXCQRL3AJCZBUXPA, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:25.438579707 (604h30m25.309662752s) [noise=true , address=45.91.116.11:4117 , piece_id=VLIHBJ7BBCKC7FC53TNNOWDXLWI5C7FOJ3RPHCXPTZM7JVI3P7PA, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:26.006186322 (604h30m25.877269367s) [noise=true , address=77.90.31.21:4878 , piece_id=CALPIEYPGBYEVEGBEMM4TVCBSAT3ELHL3S7EDMW3EMSEDKOCBVXA, seg_pos={0 0}, upload=529831]: Done uploading piece. err: <nil> Dec 30 19:59:26.006212882 (604h30m25.877295918s) [seg_pos={0 0}, upload=529831]: Segment reached optimal threshold of 13 pieces. Dec 30 19:59:26.006585312 (604h30m25.877668387s) [noise=true , address=45.91.116.10:4081 , piece_id=5OH3T6YWTPNWNAJTGNQIG4TPMWFH646OTTCQDUMYAZUHDT3HQM4A, seg_pos={0 0}, upload=529831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 30 19:59:26.006597174 (604h30m25.877680249s) [noise=true , address=45.91.116.10:4081 , piece_id=5OH3T6YWTPNWNAJTGNQIG4TPMWFH646OTTCQDUMYAZUHDT3HQM4A, seg_pos={0 0}, upload=529831]: Long tail cancellation due to optimal threshold Dec 30 19:59:26.006651678 (604h30m25.877734713s) [noise=true , address=145.79.227.49:4338 , piece_id=OYMRWST4K5KFB5PV3HZTOTK2JCV7MPZWEZBVZDHVRM7NJZBLT2NA, seg_pos={0 0}, upload=529831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 30 19:59:26.006659393 (604h30m25.877742428s) [noise=true , address=145.79.227.49:4338 , piece_id=OYMRWST4K5KFB5PV3HZTOTK2JCV7MPZWEZBVZDHVRM7NJZBLT2NA, seg_pos={0 0}, upload=529831]: Long tail cancellation due to optimal threshold Dec 30 19:59:26.006668790 (604h30m25.877751826s) [noise=true , address=145.79.227.30:4762 , piece_id=4SRJQBHND2IXLDZSS3ZH7KKYSKBTMNIYEGFF7742NERNJYJYALVQ, seg_pos={0 0}, upload=529831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 30 19:59:26.006673119 (604h30m25.877756154s) [noise=true , address=145.79.227.30:4762 , piece_id=4SRJQBHND2IXLDZSS3ZH7KKYSKBTMNIYEGFF7742NERNJYJYALVQ, seg_pos={0 0}, upload=529831]: Long tail cancellation due to optimal threshold Dec 30 19:59:26.006690051 (604h30m25.877773086s) [seg_pos={0 0}, upload=529831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 19:59:26.006786083 (604h30m25.877869119s) [upload=529831]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:59:26.006791164 (604h30m25.877874209s) [upload=529831]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:59:26.299587881 (604h30m26.170670916s) [upload=529831]: Done uploading segments... Dec 30 19:59:26.299606246 (604h30m26.170689291s) [upload=529831]: Upload finished. err: <nil> Duration: 30.00441688s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X..X.....................X......................X....XX...] duration: 20.549410266s trace: https://jaeger.dev.storj.io/trace/840262455c60921 FAIL (2025-12-30 19:59:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ, Piece ID: MNIZD24L22A677AVYFP2JTJ76YUO73X3YOJ3GBOBTEUTXLCORIEQ) context deadline exceeded; piecestore: (Node ID: 1WpVJa7SXkANKParDWYrC1cN8aLHTkUSPrRJGKQ6fvA5oDEkoa, Piece ID: LLKCRW4FFBCZJYEONAPY2PBBIUA4PHJ6VWGGARB6DEDO7KONFSXA) context deadline exceeded; piecestore: (Node ID: 1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, Piece ID: FD6B7RMU7XCL6ZI4FSG77JBTFGJNORR4CYN4RBRXKH7UGYQVCHUA) context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.40:4075 , piece_id=DZXOXPZ73ZEAWIDRN7AASB4EO4INTJHOVJ342HD5O4CYONEVAICQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.992183721 (604h27m6.863266766s) [noise=true , address=5.231.73.5:4019 , piece_id=DLKN6PTMXQGLSGHSCXJLRCTJ3QKAVXHXTE33R66HPMBMFMNZWQDA, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.992249356 (604h27m6.863332391s) [noise=true , address=77.90.31.10:4521 , piece_id=LLKCRW4FFBCZJYEONAPY2PBBIUA4PHJ6VWGGARB6DEDO7KONFSXA, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.992522767 (604h27m6.863605802s) [noise=true , address=145.79.227.6:4281 , piece_id=IJFEH45LLCFVNT3VURMQ2TWKDTKAPDLIA6DRP4ECGA6M36IHURVQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.992532625 (604h27m6.86361566s) [noise=true , address=145.79.227.39:4016 , piece_id=UK7PLDGLWUXXFBETE2T3LFBHIYHLO6Y4422XYN36AOXHCI3LBFFQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.992817888 (604h27m6.863900933s) [noise=true , address=145.79.227.22:4537 , piece_id=MKOEP4MXR2SJX4EDOIY7UDJ4G25Y5FAJ4AVAZ6KBFYKG45EDJ4TQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.993072103 (604h27m6.864155138s) [noise=true , address=77.90.31.12:4601 , piece_id=Z2TMMXC2Q3O5E4WWZ6PMZGWOBNUPSUMNKUPZCFKJOD6TV2QVDXNQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.993346675 (604h27m6.86442971s) [noise=true , address=145.79.227.49:4344 , piece_id=5DR3HZYF36VOGJ53WOZSB454C7MOWBOYE7U6IGGVTMTG4GO5DA5A, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.993630466 (604h27m6.864713501s) [noise=true , address=77.90.31.21:4891 , piece_id=KPW3XZXEW3BVHIITM4HKYFU5BEF4SVOOYRFWZENES7F7MBD2SX2A, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.993927371 (604h27m6.865010416s) [noise=true , address=145.79.227.30:4775 , piece_id=OYD6XLL2744IECKVTY5ECJUHCEWS4O7LALWZRRCNYBGER4LYVJVQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.994315810 (604h27m6.865398846s) [noise=true , address=77.90.31.17:4790 , piece_id=FD6B7RMU7XCL6ZI4FSG77JBTFGJNORR4CYN4RBRXKH7UGYQVCHUA, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.994706845 (604h27m6.86578988s) [noise=true , address=145.79.227.34:4927 , piece_id=MQZMI72BZ6JXVA3W3YHKSAYMNTPQWEPYX57TETX2QOX4BHWKZ2WA, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.995103571 (604h27m6.866186606s) [noise=true , address=45.91.116.13:4176 , piece_id=MNIZD24L22A677AVYFP2JTJ76YUO73X3YOJ3GBOBTEUTXLCORIEQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.995463917 (604h27m6.866546952s) [noise=true , address=64.49.13.139:4475 , piece_id=7JNXM7PAXURJP2X774PZ56DOFHKXMSUDJGYOXKURIRGZFONRVVOQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.996040635 (604h27m6.86712367s) [noise=true , address=145.79.227.21:4480 , piece_id=L62OLPCCYDK5EDOPPTKEITJOIXYQJZKROEP3YES6EQSHPUR7B2BA, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:08.448203770 (604h27m8.319286805s) [upload=529788]: Next returned nil segment Dec 30 19:56:08.448226784 (604h27m8.319309819s) [upload=529788]: Waiting for error group managing segments... Dec 30 19:56:08.856597951 (604h27m8.727680986s) [noise=true , address=145.79.227.22:4537 , piece_id=MKOEP4MXR2SJX4EDOIY7UDJ4G25Y5FAJ4AVAZ6KBFYKG45EDJ4TQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:08.972642035 (604h27m8.84372508s) [noise=true , address=64.49.13.139:4475 , piece_id=7JNXM7PAXURJP2X774PZ56DOFHKXMSUDJGYOXKURIRGZFONRVVOQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:08.982556104 (604h27m8.853639149s) [noise=true , address=5.231.73.5:4019 , piece_id=DLKN6PTMXQGLSGHSCXJLRCTJ3QKAVXHXTE33R66HPMBMFMNZWQDA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.017622445 (604h27m8.88870549s) [noise=true , address=77.90.31.10:4521 , piece_id=LLKCRW4FFBCZJYEONAPY2PBBIUA4PHJ6VWGGARB6DEDO7KONFSXA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.096193105 (604h27m8.96727614s) [noise=true , address=77.90.31.12:4601 , piece_id=Z2TMMXC2Q3O5E4WWZ6PMZGWOBNUPSUMNKUPZCFKJOD6TV2QVDXNQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.120549052 (604h27m8.991632097s) [noise=true , address=77.90.31.13:4655 , piece_id=ALKFNT4I4ZBK24CFWVVFHJNH4FKT6X4QO5LU5IQXLDSBNGU5FY6Q, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.121572410 (604h27m8.992655455s) [noise=true , address=77.90.31.21:4891 , piece_id=KPW3XZXEW3BVHIITM4HKYFU5BEF4SVOOYRFWZENES7F7MBD2SX2A, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.171218263 (604h27m9.042301298s) [noise=true , address=45.91.116.13:4176 , piece_id=MNIZD24L22A677AVYFP2JTJ76YUO73X3YOJ3GBOBTEUTXLCORIEQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.480613004 (604h27m9.351696049s) [noise=true , address=77.90.31.17:4790 , piece_id=FD6B7RMU7XCL6ZI4FSG77JBTFGJNORR4CYN4RBRXKH7UGYQVCHUA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.779927943 (604h27m9.651010978s) [noise=true , address=145.79.227.30:4775 , piece_id=OYD6XLL2744IECKVTY5ECJUHCEWS4O7LALWZRRCNYBGER4LYVJVQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.809206594 (604h27m9.680289649s) [noise=true , address=145.79.227.49:4344 , piece_id=5DR3HZYF36VOGJ53WOZSB454C7MOWBOYE7U6IGGVTMTG4GO5DA5A, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.882970882 (604h27m9.754053917s) [noise=true , address=145.79.227.34:4927 , piece_id=MQZMI72BZ6JXVA3W3YHKSAYMNTPQWEPYX57TETX2QOX4BHWKZ2WA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.894950363 (604h27m9.766033398s) [noise=true , address=145.79.227.44:4198 , piece_id=HTTP7BJLLNNP7BKCKC43VRRTJWESVCNS6HDMGEPLRN5GMVQHHQKA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.895290701 (604h27m9.766373736s) [noise=true , address=145.79.227.43:4189 , piece_id=SAUQY62J6PECMGHJQTSXRD6NNFWFE6A5AHVQYA3UW3DHLZUXFPIQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.895298405 (604h27m9.766381451s) [seg_pos={0 0}, upload=529788]: Segment reached optimal threshold of 14 pieces. Dec 30 19:56:09.895436178 (604h27m9.766519213s) [noise=true , address=145.79.227.21:4480 , piece_id=L62OLPCCYDK5EDOPPTKEITJOIXYQJZKROEP3YES6EQSHPUR7B2BA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qAtwKs88zrSjvRR9fvez9J9HnRBF2Ax7APfCq9BLHSFTE24a6): protocol: write tcp 188.166.225.170:59714->145.79.227.21:4480: use of closed network connection; write tcp 188.166.225.170:59714->145.79.227.21:4480: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59714->145.79.227.21:4480: use of closed network connection Dec 30 19:56:09.895442039 (604h27m9.766525074s) [noise=true , address=145.79.227.21:4480 , piece_id=L62OLPCCYDK5EDOPPTKEITJOIXYQJZKROEP3YES6EQSHPUR7B2BA, seg_pos={0 0}, upload=529788]: Long tail cancellation due to optimal threshold Dec 30 19:56:09.895651538 (604h27m9.766734573s) [noise=true , address=145.79.227.39:4016 , piece_id=UK7PLDGLWUXXFBETE2T3LFBHIYHLO6Y4422XYN36AOXHCI3LBFFQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jC98jEWn6K8jFHPA9knJeHAkuNEmcS7ThxUvbySDfbq6FmKVR): protocol: write tcp 188.166.225.170:39076->145.79.227.39:4016: use of closed network connection; write tcp 188.166.225.170:39076->145.79.227.39:4016: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39076->145.79.227.39:4016: use of closed network connection Dec 30 19:56:09.895656938 (604h27m9.766739973s) [noise=true , address=145.79.227.39:4016 , piece_id=UK7PLDGLWUXXFBETE2T3LFBHIYHLO6Y4422XYN36AOXHCI3LBFFQ, seg_pos={0 0}, upload=529788]: Long tail cancellation due to optimal threshold Dec 30 19:56:09.895753382 (604h27m9.766836417s) [noise=true , address=145.79.227.40:4075 , piece_id=DZXOXPZ73ZEAWIDRN7AASB4EO4INTJHOVJ342HD5O4CYONEVAICQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 19:56:09.895758010 (604h27m9.766841045s) [noise=true , address=145.79.227.40:4075 , piece_id=DZXOXPZ73ZEAWIDRN7AASB4EO4INTJHOVJ342HD5O4CYONEVAICQ, seg_pos={0 0}, upload=529788]: Long tail cancellation due to optimal threshold Dec 30 19:56:09.895833083 (604h27m9.766916118s) [noise=true , address=145.79.227.6:4281 , piece_id=IJFEH45LLCFVNT3VURMQ2TWKDTKAPDLIA6DRP4ECGA6M36IHURVQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hDXeAm4tomp1gAkBNjM8x77V76yiruYQmPQ3BZEYPLpJzigo2): protocol: expected piece hash; context canceled; EOF Dec 30 19:56:09.895837542 (604h27m9.766920577s) [noise=true , address=145.79.227.6:4281 , piece_id=IJFEH45LLCFVNT3VURMQ2TWKDTKAPDLIA6DRP4ECGA6M36IHURVQ, seg_pos={0 0}, upload=529788]: Long tail cancellation due to optimal threshold Dec 30 19:56:09.895850146 (604h27m9.766933191s) [seg_pos={0 0}, upload=529788]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 19:56:09.895926181 (604h27m9.767009216s) [upload=529788]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:56:09.895933535 (604h27m9.76701657s) [upload=529788]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:56:10.230726588 (604h27m10.101809633s) [upload=529788]: Done uploading segments... Dec 30 19:56:10.230750393 (604h27m10.101833439s) [upload=529788]: Upload finished. err: <nil> Duration: 20.54943861s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X..X.....................X......................X....XX..] duration: 33.01307931s trace: https://jaeger.dev.storj.io/trace/840262455c60921 FAIL (2025-12-30 19:58:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ, Piece ID: MNIZD24L22A677AVYFP2JTJ76YUO73X3YOJ3GBOBTEUTXLCORIEQ) context deadline exceeded; piecestore: (Node ID: 1WpVJa7SXkANKParDWYrC1cN8aLHTkUSPrRJGKQ6fvA5oDEkoa, Piece ID: LLKCRW4FFBCZJYEONAPY2PBBIUA4PHJ6VWGGARB6DEDO7KONFSXA) context deadline exceeded; piecestore: (Node ID: 1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, Piece ID: FD6B7RMU7XCL6ZI4FSG77JBTFGJNORR4CYN4RBRXKH7UGYQVCHUA) context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.40:4075 , piece_id=DZXOXPZ73ZEAWIDRN7AASB4EO4INTJHOVJ342HD5O4CYONEVAICQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.992183721 (604h27m6.863266766s) [noise=true , address=5.231.73.5:4019 , piece_id=DLKN6PTMXQGLSGHSCXJLRCTJ3QKAVXHXTE33R66HPMBMFMNZWQDA, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.992249356 (604h27m6.863332391s) [noise=true , address=77.90.31.10:4521 , piece_id=LLKCRW4FFBCZJYEONAPY2PBBIUA4PHJ6VWGGARB6DEDO7KONFSXA, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.992522767 (604h27m6.863605802s) [noise=true , address=145.79.227.6:4281 , piece_id=IJFEH45LLCFVNT3VURMQ2TWKDTKAPDLIA6DRP4ECGA6M36IHURVQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.992532625 (604h27m6.86361566s) [noise=true , address=145.79.227.39:4016 , piece_id=UK7PLDGLWUXXFBETE2T3LFBHIYHLO6Y4422XYN36AOXHCI3LBFFQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.992817888 (604h27m6.863900933s) [noise=true , address=145.79.227.22:4537 , piece_id=MKOEP4MXR2SJX4EDOIY7UDJ4G25Y5FAJ4AVAZ6KBFYKG45EDJ4TQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.993072103 (604h27m6.864155138s) [noise=true , address=77.90.31.12:4601 , piece_id=Z2TMMXC2Q3O5E4WWZ6PMZGWOBNUPSUMNKUPZCFKJOD6TV2QVDXNQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.993346675 (604h27m6.86442971s) [noise=true , address=145.79.227.49:4344 , piece_id=5DR3HZYF36VOGJ53WOZSB454C7MOWBOYE7U6IGGVTMTG4GO5DA5A, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.993630466 (604h27m6.864713501s) [noise=true , address=77.90.31.21:4891 , piece_id=KPW3XZXEW3BVHIITM4HKYFU5BEF4SVOOYRFWZENES7F7MBD2SX2A, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.993927371 (604h27m6.865010416s) [noise=true , address=145.79.227.30:4775 , piece_id=OYD6XLL2744IECKVTY5ECJUHCEWS4O7LALWZRRCNYBGER4LYVJVQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.994315810 (604h27m6.865398846s) [noise=true , address=77.90.31.17:4790 , piece_id=FD6B7RMU7XCL6ZI4FSG77JBTFGJNORR4CYN4RBRXKH7UGYQVCHUA, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.994706845 (604h27m6.86578988s) [noise=true , address=145.79.227.34:4927 , piece_id=MQZMI72BZ6JXVA3W3YHKSAYMNTPQWEPYX57TETX2QOX4BHWKZ2WA, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.995103571 (604h27m6.866186606s) [noise=true , address=45.91.116.13:4176 , piece_id=MNIZD24L22A677AVYFP2JTJ76YUO73X3YOJ3GBOBTEUTXLCORIEQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.995463917 (604h27m6.866546952s) [noise=true , address=64.49.13.139:4475 , piece_id=7JNXM7PAXURJP2X774PZ56DOFHKXMSUDJGYOXKURIRGZFONRVVOQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.996040635 (604h27m6.86712367s) [noise=true , address=145.79.227.21:4480 , piece_id=L62OLPCCYDK5EDOPPTKEITJOIXYQJZKROEP3YES6EQSHPUR7B2BA, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:08.448203770 (604h27m8.319286805s) [upload=529788]: Next returned nil segment Dec 30 19:56:08.448226784 (604h27m8.319309819s) [upload=529788]: Waiting for error group managing segments... Dec 30 19:56:08.856597951 (604h27m8.727680986s) [noise=true , address=145.79.227.22:4537 , piece_id=MKOEP4MXR2SJX4EDOIY7UDJ4G25Y5FAJ4AVAZ6KBFYKG45EDJ4TQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:08.972642035 (604h27m8.84372508s) [noise=true , address=64.49.13.139:4475 , piece_id=7JNXM7PAXURJP2X774PZ56DOFHKXMSUDJGYOXKURIRGZFONRVVOQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:08.982556104 (604h27m8.853639149s) [noise=true , address=5.231.73.5:4019 , piece_id=DLKN6PTMXQGLSGHSCXJLRCTJ3QKAVXHXTE33R66HPMBMFMNZWQDA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.017622445 (604h27m8.88870549s) [noise=true , address=77.90.31.10:4521 , piece_id=LLKCRW4FFBCZJYEONAPY2PBBIUA4PHJ6VWGGARB6DEDO7KONFSXA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.096193105 (604h27m8.96727614s) [noise=true , address=77.90.31.12:4601 , piece_id=Z2TMMXC2Q3O5E4WWZ6PMZGWOBNUPSUMNKUPZCFKJOD6TV2QVDXNQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.120549052 (604h27m8.991632097s) [noise=true , address=77.90.31.13:4655 , piece_id=ALKFNT4I4ZBK24CFWVVFHJNH4FKT6X4QO5LU5IQXLDSBNGU5FY6Q, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.121572410 (604h27m8.992655455s) [noise=true , address=77.90.31.21:4891 , piece_id=KPW3XZXEW3BVHIITM4HKYFU5BEF4SVOOYRFWZENES7F7MBD2SX2A, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.171218263 (604h27m9.042301298s) [noise=true , address=45.91.116.13:4176 , piece_id=MNIZD24L22A677AVYFP2JTJ76YUO73X3YOJ3GBOBTEUTXLCORIEQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.480613004 (604h27m9.351696049s) [noise=true , address=77.90.31.17:4790 , piece_id=FD6B7RMU7XCL6ZI4FSG77JBTFGJNORR4CYN4RBRXKH7UGYQVCHUA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.779927943 (604h27m9.651010978s) [noise=true , address=145.79.227.30:4775 , piece_id=OYD6XLL2744IECKVTY5ECJUHCEWS4O7LALWZRRCNYBGER4LYVJVQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.809206594 (604h27m9.680289649s) [noise=true , address=145.79.227.49:4344 , piece_id=5DR3HZYF36VOGJ53WOZSB454C7MOWBOYE7U6IGGVTMTG4GO5DA5A, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.882970882 (604h27m9.754053917s) [noise=true , address=145.79.227.34:4927 , piece_id=MQZMI72BZ6JXVA3W3YHKSAYMNTPQWEPYX57TETX2QOX4BHWKZ2WA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.894950363 (604h27m9.766033398s) [noise=true , address=145.79.227.44:4198 , piece_id=HTTP7BJLLNNP7BKCKC43VRRTJWESVCNS6HDMGEPLRN5GMVQHHQKA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.895290701 (604h27m9.766373736s) [noise=true , address=145.79.227.43:4189 , piece_id=SAUQY62J6PECMGHJQTSXRD6NNFWFE6A5AHVQYA3UW3DHLZUXFPIQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.895298405 (604h27m9.766381451s) [seg_pos={0 0}, upload=529788]: Segment reached optimal threshold of 14 pieces. Dec 30 19:56:09.895436178 (604h27m9.766519213s) [noise=true , address=145.79.227.21:4480 , piece_id=L62OLPCCYDK5EDOPPTKEITJOIXYQJZKROEP3YES6EQSHPUR7B2BA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qAtwKs88zrSjvRR9fvez9J9HnRBF2Ax7APfCq9BLHSFTE24a6): protocol: write tcp 188.166.225.170:59714->145.79.227.21:4480: use of closed network connection; write tcp 188.166.225.170:59714->145.79.227.21:4480: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59714->145.79.227.21:4480: use of closed network connection Dec 30 19:56:09.895442039 (604h27m9.766525074s) [noise=true , address=145.79.227.21:4480 , piece_id=L62OLPCCYDK5EDOPPTKEITJOIXYQJZKROEP3YES6EQSHPUR7B2BA, seg_pos={0 0}, upload=529788]: Long tail cancellation due to optimal threshold Dec 30 19:56:09.895651538 (604h27m9.766734573s) [noise=true , address=145.79.227.39:4016 , piece_id=UK7PLDGLWUXXFBETE2T3LFBHIYHLO6Y4422XYN36AOXHCI3LBFFQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jC98jEWn6K8jFHPA9knJeHAkuNEmcS7ThxUvbySDfbq6FmKVR): protocol: write tcp 188.166.225.170:39076->145.79.227.39:4016: use of closed network connection; write tcp 188.166.225.170:39076->145.79.227.39:4016: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39076->145.79.227.39:4016: use of closed network connection Dec 30 19:56:09.895656938 (604h27m9.766739973s) [noise=true , address=145.79.227.39:4016 , piece_id=UK7PLDGLWUXXFBETE2T3LFBHIYHLO6Y4422XYN36AOXHCI3LBFFQ, seg_pos={0 0}, upload=529788]: Long tail cancellation due to optimal threshold Dec 30 19:56:09.895753382 (604h27m9.766836417s) [noise=true , address=145.79.227.40:4075 , piece_id=DZXOXPZ73ZEAWIDRN7AASB4EO4INTJHOVJ342HD5O4CYONEVAICQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 19:56:09.895758010 (604h27m9.766841045s) [noise=true , address=145.79.227.40:4075 , piece_id=DZXOXPZ73ZEAWIDRN7AASB4EO4INTJHOVJ342HD5O4CYONEVAICQ, seg_pos={0 0}, upload=529788]: Long tail cancellation due to optimal threshold Dec 30 19:56:09.895833083 (604h27m9.766916118s) [noise=true , address=145.79.227.6:4281 , piece_id=IJFEH45LLCFVNT3VURMQ2TWKDTKAPDLIA6DRP4ECGA6M36IHURVQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hDXeAm4tomp1gAkBNjM8x77V76yiruYQmPQ3BZEYPLpJzigo2): protocol: expected piece hash; context canceled; EOF Dec 30 19:56:09.895837542 (604h27m9.766920577s) [noise=true , address=145.79.227.6:4281 , piece_id=IJFEH45LLCFVNT3VURMQ2TWKDTKAPDLIA6DRP4ECGA6M36IHURVQ, seg_pos={0 0}, upload=529788]: Long tail cancellation due to optimal threshold Dec 30 19:56:09.895850146 (604h27m9.766933191s) [seg_pos={0 0}, upload=529788]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 19:56:09.895926181 (604h27m9.767009216s) [upload=529788]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:56:09.895933535 (604h27m9.76701657s) [upload=529788]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:56:10.230726588 (604h27m10.101809633s) [upload=529788]: Done uploading segments... Dec 30 19:56:10.230750393 (604h27m10.101833439s) [upload=529788]: Upload finished. err: <nil> Duration: 33.013109247s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.XX...X..X.....XX.X.....X.....X.................X...X.X.X...] duration: 11.444244129s trace: https://jaeger.dev.storj.io/trace/796612bf4f96ea62 FAIL (2025-12-30 19:58:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: 7OTARUUJ5SMUPWNWGJU7UPZ4BH6EXUNBOLHVDRQKCPS26VRVNQTQ) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: E2WQ2IE4XEHNLT5ZOI36XBVWKUTT2XKQCMMCJHGISGW5PM23OSPA) context deadline exceeded ======LOGS====== Dec 30 19:56:36.978167026 (604h27m36.849250061s) [upload=529794]: Starting upload Dec 30 19:56:36.978278318 (604h27m36.849361373s) [upload=529794]: Done starting upload Dec 30 19:56:36.978339805 (604h27m36.84942284s) [upload=529794]: Uploading segments... Dec 30 19:56:36.978599078 (604h27m36.849682113s) [upload=529794]: Got next segment. Inline: false Dec 30 19:56:36.978635257 (604h27m36.849718292s) [upload=529794]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 19:56:36.978640226 (604h27m36.849723261s) [upload=529794]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 19:56:37.224970965 (604h27m37.096054s) [seg_pos={0 0}, upload=529794]: Begin upload segment... Dec 30 19:56:37.225024648 (604h27m37.096107683s) [seg_pos={0 0}, upload=529794]: Done begin upload segment. Dec 30 19:56:37.225055818 (604h27m37.096138853s) [noise=true , address=145.79.227.45:4229 , piece_id=EXTZJT63CIG6MYSAOTQ6FS24CWQ26P5SABM2XCGLGO5S6KQYFSLA, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.225073160 (604h27m37.096156205s) [noise=true , address=45.91.116.5:4298 , piece_id=QZULXYQVEFQBIDBZBDLT7Z67X5LK4L4JNNXYG2UYKWMIHAJDEPDQ, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.225903902 (604h27m37.096986967s) [noise=true , address=145.79.227.17:4970 , piece_id=VN3JRKKNAGSJK6QK3LJRCM553O4POX6K7CL2OZPT33WVC2W3Y3UA, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.226564250 (604h27m37.097647295s) [noise=true , address=145.79.227.18:4007 , piece_id=KPR5I4W6SHDPL5H6ICSQYAV6IMJ6SBT6HBLVZPCOWNBJB24L2YJA, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.227046768 (604h27m37.098129804s) [noise=true , address=145.79.227.39:4015 , piece_id=AMGJEADOFORWWDEB5YWQKWSBER5F4G4P2XY3VOW66LCFVVYPZ5HQ, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.227690133 (604h27m37.098773168s) [noise=true , address=77.90.31.17:4770 , piece_id=PXB64XW2RPG3Y7EJFF4VSH2SVHBLGNCQSROV424VVVH7E7XEOIVA, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.228146492 (604h27m37.099229547s) [noise=true , address=145.79.227.32:4835 , piece_id=BGKMICBP2KWG6PO4AUSXNN5QKKILJ3THBENIXREYQCIWVVAIRQWQ, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.228545893 (604h27m37.099628928s) [noise=true , address=45.91.116.13:4155 , piece_id=CKYZYLMMWE5BA25VSXZY7XPROPFU6NPKXQPDJ7FMUASFNSYLO2ZA, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.228926797 (604h27m37.100009832s) [noise=true , address=77.90.31.15:4697 , piece_id=E2WQ2IE4XEHNLT5ZOI36XBVWKUTT2XKQCMMCJHGISGW5PM23OSPA, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.229268368 (604h27m37.100351403s) [noise=true , address=77.90.31.14:4664 , piece_id=7OTARUUJ5SMUPWNWGJU7UPZ4BH6EXUNBOLHVDRQKCPS26VRVNQTQ, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.229587335 (604h27m37.10067037s) [noise=true , address=145.79.227.8:4330 , piece_id=5VDJ3GBK43DHSGSOAS3P74PHDQUHCRTX6ADXXYNKKYFBTSLXC6UA, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.229993659 (604h27m37.101076694s) [noise=true , address=145.79.227.38:4979 , piece_id=K5YWQA3JXONYZGPNBW27ZCWAAIXDYXTHCYXXVHI4QATMTXHNXGHQ, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.230365578 (604h27m37.101448613s) [noise=true , address=77.90.31.19:4841 , piece_id=SL2XF2NPCJ4REJ7FMH4KQ5R7URDBBXR2IC4GKWEB5XWUHXWOVC7A, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.231935166 (604h27m37.103018201s) [noise=true , address=45.91.116.8:4046 , piece_id=FJRJS3ZJTEXIPK4TKCTLVSCE6RYFFKY64KWLYHK6IW52E7OM7NQQ, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.232619950 (604h27m37.103702995s) [noise=true , address=45.91.116.9:4264 , piece_id=XEJJGSR3H2PEXM7YWNCUVZYXGWQ63FL7MBSQG3FSN7IWRLAWWUIA, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.233080628 (604h27m37.104163663s) [noise=true , address=145.79.227.4:4186 , piece_id=CUF5JYTXABJ5VOWPWCUCSZXBDHLY5EM2JGES4HRXDCHVY2IAY3TQ, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:38.941787109 (604h27m38.812870144s) [upload=529794]: Next returned nil segment Dec 30 19:56:38.941854097 (604h27m38.812937162s) [upload=529794]: Waiting for error group managing segments... Dec 30 19:56:39.223299549 (604h27m39.094382584s) [noise=true , address=145.79.227.4:4186 , piece_id=CUF5JYTXABJ5VOWPWCUCSZXBDHLY5EM2JGES4HRXDCHVY2IAY3TQ, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.229871974 (604h27m39.100955009s) [noise=true , address=145.79.227.38:4979 , piece_id=K5YWQA3JXONYZGPNBW27ZCWAAIXDYXTHCYXXVHI4QATMTXHNXGHQ, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.233639237 (604h27m39.104722282s) [noise=true , address=145.79.227.45:4229 , piece_id=EXTZJT63CIG6MYSAOTQ6FS24CWQ26P5SABM2XCGLGO5S6KQYFSLA, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.234939713 (604h27m39.106022748s) [noise=true , address=145.79.227.8:4330 , piece_id=5VDJ3GBK43DHSGSOAS3P74PHDQUHCRTX6ADXXYNKKYFBTSLXC6UA, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.241257905 (604h27m39.11234094s) [noise=true , address=145.79.227.39:4015 , piece_id=AMGJEADOFORWWDEB5YWQKWSBER5F4G4P2XY3VOW66LCFVVYPZ5HQ, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.249599698 (604h27m39.120682733s) [noise=true , address=145.79.227.18:4007 , piece_id=KPR5I4W6SHDPL5H6ICSQYAV6IMJ6SBT6HBLVZPCOWNBJB24L2YJA, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.267283862 (604h27m39.138366897s) [noise=true , address=145.79.227.17:4970 , piece_id=VN3JRKKNAGSJK6QK3LJRCM553O4POX6K7CL2OZPT33WVC2W3Y3UA, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.316366802 (604h27m39.187449837s) [noise=true , address=145.79.227.32:4835 , piece_id=BGKMICBP2KWG6PO4AUSXNN5QKKILJ3THBENIXREYQCIWVVAIRQWQ, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.453863376 (604h27m39.324946411s) [noise=true , address=77.90.31.17:4770 , piece_id=PXB64XW2RPG3Y7EJFF4VSH2SVHBLGNCQSROV424VVVH7E7XEOIVA, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.489946233 (604h27m39.361029268s) [noise=true , address=77.90.31.15:4697 , piece_id=E2WQ2IE4XEHNLT5ZOI36XBVWKUTT2XKQCMMCJHGISGW5PM23OSPA, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.492675480 (604h27m39.363758525s) [noise=true , address=77.90.31.14:4664 , piece_id=7OTARUUJ5SMUPWNWGJU7UPZ4BH6EXUNBOLHVDRQKCPS26VRVNQTQ, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.580589390 (604h27m39.451672435s) [noise=true , address=45.91.116.13:4155 , piece_id=CKYZYLMMWE5BA25VSXZY7XPROPFU6NPKXQPDJ7FMUASFNSYLO2ZA, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.580711223 (604h27m39.451794268s) [noise=true , address=45.91.116.9:4264 , piece_id=XEJJGSR3H2PEXM7YWNCUVZYXGWQ63FL7MBSQG3FSN7IWRLAWWUIA, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.580734668 (604h27m39.451817713s) [seg_pos={0 0}, upload=529794]: Segment reached optimal threshold of 13 pieces. Dec 30 19:56:39.580946010 (604h27m39.452029055s) [noise=true , address=45.91.116.5:4298 , piece_id=QZULXYQVEFQBIDBZBDLT7Z67X5LK4L4JNNXYG2UYKWMIHAJDEPDQ, seg_pos={0 0}, upload=529794]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 30 19:56:39.580964675 (604h27m39.45204771s) [noise=true , address=45.91.116.5:4298 , piece_id=QZULXYQVEFQBIDBZBDLT7Z67X5LK4L4JNNXYG2UYKWMIHAJDEPDQ, seg_pos={0 0}, upload=529794]: Long tail cancellation due to optimal threshold Dec 30 19:56:39.581009039 (604h27m39.452092095s) [noise=true , address=45.91.116.8:4046 , piece_id=FJRJS3ZJTEXIPK4TKCTLVSCE6RYFFKY64KWLYHK6IW52E7OM7NQQ, seg_pos={0 0}, upload=529794]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 30 19:56:39.581019841 (604h27m39.452102896s) [noise=true , address=45.91.116.8:4046 , piece_id=FJRJS3ZJTEXIPK4TKCTLVSCE6RYFFKY64KWLYHK6IW52E7OM7NQQ, seg_pos={0 0}, upload=529794]: Long tail cancellation due to optimal threshold Dec 30 19:56:39.581069174 (604h27m39.452152209s) [noise=true , address=77.90.31.19:4841 , piece_id=SL2XF2NPCJ4REJ7FMH4KQ5R7URDBBXR2IC4GKWEB5XWUHXWOVC7A, seg_pos={0 0}, upload=529794]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Dec 30 19:56:39.581076298 (604h27m39.452159333s) [noise=true , address=77.90.31.19:4841 , piece_id=SL2XF2NPCJ4REJ7FMH4KQ5R7URDBBXR2IC4GKWEB5XWUHXWOVC7A, seg_pos={0 0}, upload=529794]: Long tail cancellation due to optimal threshold Dec 30 19:56:39.581097528 (604h27m39.452180563s) [seg_pos={0 0}, upload=529794]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 19:56:39.581197789 (604h27m39.452280824s) [upload=529794]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:56:39.581204532 (604h27m39.452287567s) [upload=529794]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:56:39.872427473 (604h27m39.743510508s) [upload=529794]: Done uploading segments... Dec 30 19:56:39.872446579 (604h27m39.743529624s) [upload=529794]: Upload finished. err: <nil> Duration: 11.444274085s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..XX...X..X.....XX.X.....X.....X.................X...X.X.X..] duration: 6.561386466s trace: https://jaeger.dev.storj.io/trace/796612bf4f96ea62 FAIL (2025-12-30 19:57:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: 7OTARUUJ5SMUPWNWGJU7UPZ4BH6EXUNBOLHVDRQKCPS26VRVNQTQ) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: E2WQ2IE4XEHNLT5ZOI36XBVWKUTT2XKQCMMCJHGISGW5PM23OSPA) context deadline exceeded ======LOGS====== Dec 30 19:56:36.978167026 (604h27m36.849250061s) [upload=529794]: Starting upload Dec 30 19:56:36.978278318 (604h27m36.849361373s) [upload=529794]: Done starting upload Dec 30 19:56:36.978339805 (604h27m36.84942284s) [upload=529794]: Uploading segments... Dec 30 19:56:36.978599078 (604h27m36.849682113s) [upload=529794]: Got next segment. Inline: false Dec 30 19:56:36.978635257 (604h27m36.849718292s) [upload=529794]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 19:56:36.978640226 (604h27m36.849723261s) [upload=529794]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 19:56:37.224970965 (604h27m37.096054s) [seg_pos={0 0}, upload=529794]: Begin upload segment... Dec 30 19:56:37.225024648 (604h27m37.096107683s) [seg_pos={0 0}, upload=529794]: Done begin upload segment. Dec 30 19:56:37.225055818 (604h27m37.096138853s) [noise=true , address=145.79.227.45:4229 , piece_id=EXTZJT63CIG6MYSAOTQ6FS24CWQ26P5SABM2XCGLGO5S6KQYFSLA, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.225073160 (604h27m37.096156205s) [noise=true , address=45.91.116.5:4298 , piece_id=QZULXYQVEFQBIDBZBDLT7Z67X5LK4L4JNNXYG2UYKWMIHAJDEPDQ, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.225903902 (604h27m37.096986967s) [noise=true , address=145.79.227.17:4970 , piece_id=VN3JRKKNAGSJK6QK3LJRCM553O4POX6K7CL2OZPT33WVC2W3Y3UA, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.226564250 (604h27m37.097647295s) [noise=true , address=145.79.227.18:4007 , piece_id=KPR5I4W6SHDPL5H6ICSQYAV6IMJ6SBT6HBLVZPCOWNBJB24L2YJA, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.227046768 (604h27m37.098129804s) [noise=true , address=145.79.227.39:4015 , piece_id=AMGJEADOFORWWDEB5YWQKWSBER5F4G4P2XY3VOW66LCFVVYPZ5HQ, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.227690133 (604h27m37.098773168s) [noise=true , address=77.90.31.17:4770 , piece_id=PXB64XW2RPG3Y7EJFF4VSH2SVHBLGNCQSROV424VVVH7E7XEOIVA, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.228146492 (604h27m37.099229547s) [noise=true , address=145.79.227.32:4835 , piece_id=BGKMICBP2KWG6PO4AUSXNN5QKKILJ3THBENIXREYQCIWVVAIRQWQ, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.228545893 (604h27m37.099628928s) [noise=true , address=45.91.116.13:4155 , piece_id=CKYZYLMMWE5BA25VSXZY7XPROPFU6NPKXQPDJ7FMUASFNSYLO2ZA, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.228926797 (604h27m37.100009832s) [noise=true , address=77.90.31.15:4697 , piece_id=E2WQ2IE4XEHNLT5ZOI36XBVWKUTT2XKQCMMCJHGISGW5PM23OSPA, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.229268368 (604h27m37.100351403s) [noise=true , address=77.90.31.14:4664 , piece_id=7OTARUUJ5SMUPWNWGJU7UPZ4BH6EXUNBOLHVDRQKCPS26VRVNQTQ, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.229587335 (604h27m37.10067037s) [noise=true , address=145.79.227.8:4330 , piece_id=5VDJ3GBK43DHSGSOAS3P74PHDQUHCRTX6ADXXYNKKYFBTSLXC6UA, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.229993659 (604h27m37.101076694s) [noise=true , address=145.79.227.38:4979 , piece_id=K5YWQA3JXONYZGPNBW27ZCWAAIXDYXTHCYXXVHI4QATMTXHNXGHQ, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.230365578 (604h27m37.101448613s) [noise=true , address=77.90.31.19:4841 , piece_id=SL2XF2NPCJ4REJ7FMH4KQ5R7URDBBXR2IC4GKWEB5XWUHXWOVC7A, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.231935166 (604h27m37.103018201s) [noise=true , address=45.91.116.8:4046 , piece_id=FJRJS3ZJTEXIPK4TKCTLVSCE6RYFFKY64KWLYHK6IW52E7OM7NQQ, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.232619950 (604h27m37.103702995s) [noise=true , address=45.91.116.9:4264 , piece_id=XEJJGSR3H2PEXM7YWNCUVZYXGWQ63FL7MBSQG3FSN7IWRLAWWUIA, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.233080628 (604h27m37.104163663s) [noise=true , address=145.79.227.4:4186 , piece_id=CUF5JYTXABJ5VOWPWCUCSZXBDHLY5EM2JGES4HRXDCHVY2IAY3TQ, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:38.941787109 (604h27m38.812870144s) [upload=529794]: Next returned nil segment Dec 30 19:56:38.941854097 (604h27m38.812937162s) [upload=529794]: Waiting for error group managing segments... Dec 30 19:56:39.223299549 (604h27m39.094382584s) [noise=true , address=145.79.227.4:4186 , piece_id=CUF5JYTXABJ5VOWPWCUCSZXBDHLY5EM2JGES4HRXDCHVY2IAY3TQ, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.229871974 (604h27m39.100955009s) [noise=true , address=145.79.227.38:4979 , piece_id=K5YWQA3JXONYZGPNBW27ZCWAAIXDYXTHCYXXVHI4QATMTXHNXGHQ, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.233639237 (604h27m39.104722282s) [noise=true , address=145.79.227.45:4229 , piece_id=EXTZJT63CIG6MYSAOTQ6FS24CWQ26P5SABM2XCGLGO5S6KQYFSLA, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.234939713 (604h27m39.106022748s) [noise=true , address=145.79.227.8:4330 , piece_id=5VDJ3GBK43DHSGSOAS3P74PHDQUHCRTX6ADXXYNKKYFBTSLXC6UA, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.241257905 (604h27m39.11234094s) [noise=true , address=145.79.227.39:4015 , piece_id=AMGJEADOFORWWDEB5YWQKWSBER5F4G4P2XY3VOW66LCFVVYPZ5HQ, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.249599698 (604h27m39.120682733s) [noise=true , address=145.79.227.18:4007 , piece_id=KPR5I4W6SHDPL5H6ICSQYAV6IMJ6SBT6HBLVZPCOWNBJB24L2YJA, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.267283862 (604h27m39.138366897s) [noise=true , address=145.79.227.17:4970 , piece_id=VN3JRKKNAGSJK6QK3LJRCM553O4POX6K7CL2OZPT33WVC2W3Y3UA, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.316366802 (604h27m39.187449837s) [noise=true , address=145.79.227.32:4835 , piece_id=BGKMICBP2KWG6PO4AUSXNN5QKKILJ3THBENIXREYQCIWVVAIRQWQ, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.453863376 (604h27m39.324946411s) [noise=true , address=77.90.31.17:4770 , piece_id=PXB64XW2RPG3Y7EJFF4VSH2SVHBLGNCQSROV424VVVH7E7XEOIVA, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.489946233 (604h27m39.361029268s) [noise=true , address=77.90.31.15:4697 , piece_id=E2WQ2IE4XEHNLT5ZOI36XBVWKUTT2XKQCMMCJHGISGW5PM23OSPA, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.492675480 (604h27m39.363758525s) [noise=true , address=77.90.31.14:4664 , piece_id=7OTARUUJ5SMUPWNWGJU7UPZ4BH6EXUNBOLHVDRQKCPS26VRVNQTQ, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.580589390 (604h27m39.451672435s) [noise=true , address=45.91.116.13:4155 , piece_id=CKYZYLMMWE5BA25VSXZY7XPROPFU6NPKXQPDJ7FMUASFNSYLO2ZA, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.580711223 (604h27m39.451794268s) [noise=true , address=45.91.116.9:4264 , piece_id=XEJJGSR3H2PEXM7YWNCUVZYXGWQ63FL7MBSQG3FSN7IWRLAWWUIA, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.580734668 (604h27m39.451817713s) [seg_pos={0 0}, upload=529794]: Segment reached optimal threshold of 13 pieces. Dec 30 19:56:39.580946010 (604h27m39.452029055s) [noise=true , address=45.91.116.5:4298 , piece_id=QZULXYQVEFQBIDBZBDLT7Z67X5LK4L4JNNXYG2UYKWMIHAJDEPDQ, seg_pos={0 0}, upload=529794]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 30 19:56:39.580964675 (604h27m39.45204771s) [noise=true , address=45.91.116.5:4298 , piece_id=QZULXYQVEFQBIDBZBDLT7Z67X5LK4L4JNNXYG2UYKWMIHAJDEPDQ, seg_pos={0 0}, upload=529794]: Long tail cancellation due to optimal threshold Dec 30 19:56:39.581009039 (604h27m39.452092095s) [noise=true , address=45.91.116.8:4046 , piece_id=FJRJS3ZJTEXIPK4TKCTLVSCE6RYFFKY64KWLYHK6IW52E7OM7NQQ, seg_pos={0 0}, upload=529794]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 30 19:56:39.581019841 (604h27m39.452102896s) [noise=true , address=45.91.116.8:4046 , piece_id=FJRJS3ZJTEXIPK4TKCTLVSCE6RYFFKY64KWLYHK6IW52E7OM7NQQ, seg_pos={0 0}, upload=529794]: Long tail cancellation due to optimal threshold Dec 30 19:56:39.581069174 (604h27m39.452152209s) [noise=true , address=77.90.31.19:4841 , piece_id=SL2XF2NPCJ4REJ7FMH4KQ5R7URDBBXR2IC4GKWEB5XWUHXWOVC7A, seg_pos={0 0}, upload=529794]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Dec 30 19:56:39.581076298 (604h27m39.452159333s) [noise=true , address=77.90.31.19:4841 , piece_id=SL2XF2NPCJ4REJ7FMH4KQ5R7URDBBXR2IC4GKWEB5XWUHXWOVC7A, seg_pos={0 0}, upload=529794]: Long tail cancellation due to optimal threshold Dec 30 19:56:39.581097528 (604h27m39.452180563s) [seg_pos={0 0}, upload=529794]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 19:56:39.581197789 (604h27m39.452280824s) [upload=529794]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:56:39.581204532 (604h27m39.452287567s) [upload=529794]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:56:39.872427473 (604h27m39.743510508s) [upload=529794]: Done uploading segments... Dec 30 19:56:39.872446579 (604h27m39.743529624s) [upload=529794]: Upload finished. err: <nil> Duration: 6.561415141s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X..XX...X..X.....XX.X.....X.....X.................X...X.X.X.] duration: 29.173514249s trace: https://jaeger.dev.storj.io/trace/796612bf4f96ea62 FAIL (2025-12-30 19:57:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: 7OTARUUJ5SMUPWNWGJU7UPZ4BH6EXUNBOLHVDRQKCPS26VRVNQTQ) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: E2WQ2IE4XEHNLT5ZOI36XBVWKUTT2XKQCMMCJHGISGW5PM23OSPA) context deadline exceeded ======LOGS====== Dec 30 19:56:36.978167026 (604h27m36.849250061s) [upload=529794]: Starting upload Dec 30 19:56:36.978278318 (604h27m36.849361373s) [upload=529794]: Done starting upload Dec 30 19:56:36.978339805 (604h27m36.84942284s) [upload=529794]: Uploading segments... Dec 30 19:56:36.978599078 (604h27m36.849682113s) [upload=529794]: Got next segment. Inline: false Dec 30 19:56:36.978635257 (604h27m36.849718292s) [upload=529794]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 19:56:36.978640226 (604h27m36.849723261s) [upload=529794]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 19:56:37.224970965 (604h27m37.096054s) [seg_pos={0 0}, upload=529794]: Begin upload segment... Dec 30 19:56:37.225024648 (604h27m37.096107683s) [seg_pos={0 0}, upload=529794]: Done begin upload segment. Dec 30 19:56:37.225055818 (604h27m37.096138853s) [noise=true , address=145.79.227.45:4229 , piece_id=EXTZJT63CIG6MYSAOTQ6FS24CWQ26P5SABM2XCGLGO5S6KQYFSLA, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.225073160 (604h27m37.096156205s) [noise=true , address=45.91.116.5:4298 , piece_id=QZULXYQVEFQBIDBZBDLT7Z67X5LK4L4JNNXYG2UYKWMIHAJDEPDQ, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.225903902 (604h27m37.096986967s) [noise=true , address=145.79.227.17:4970 , piece_id=VN3JRKKNAGSJK6QK3LJRCM553O4POX6K7CL2OZPT33WVC2W3Y3UA, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.226564250 (604h27m37.097647295s) [noise=true , address=145.79.227.18:4007 , piece_id=KPR5I4W6SHDPL5H6ICSQYAV6IMJ6SBT6HBLVZPCOWNBJB24L2YJA, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.227046768 (604h27m37.098129804s) [noise=true , address=145.79.227.39:4015 , piece_id=AMGJEADOFORWWDEB5YWQKWSBER5F4G4P2XY3VOW66LCFVVYPZ5HQ, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.227690133 (604h27m37.098773168s) [noise=true , address=77.90.31.17:4770 , piece_id=PXB64XW2RPG3Y7EJFF4VSH2SVHBLGNCQSROV424VVVH7E7XEOIVA, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.228146492 (604h27m37.099229547s) [noise=true , address=145.79.227.32:4835 , piece_id=BGKMICBP2KWG6PO4AUSXNN5QKKILJ3THBENIXREYQCIWVVAIRQWQ, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.228545893 (604h27m37.099628928s) [noise=true , address=45.91.116.13:4155 , piece_id=CKYZYLMMWE5BA25VSXZY7XPROPFU6NPKXQPDJ7FMUASFNSYLO2ZA, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.228926797 (604h27m37.100009832s) [noise=true , address=77.90.31.15:4697 , piece_id=E2WQ2IE4XEHNLT5ZOI36XBVWKUTT2XKQCMMCJHGISGW5PM23OSPA, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.229268368 (604h27m37.100351403s) [noise=true , address=77.90.31.14:4664 , piece_id=7OTARUUJ5SMUPWNWGJU7UPZ4BH6EXUNBOLHVDRQKCPS26VRVNQTQ, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.229587335 (604h27m37.10067037s) [noise=true , address=145.79.227.8:4330 , piece_id=5VDJ3GBK43DHSGSOAS3P74PHDQUHCRTX6ADXXYNKKYFBTSLXC6UA, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.229993659 (604h27m37.101076694s) [noise=true , address=145.79.227.38:4979 , piece_id=K5YWQA3JXONYZGPNBW27ZCWAAIXDYXTHCYXXVHI4QATMTXHNXGHQ, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.230365578 (604h27m37.101448613s) [noise=true , address=77.90.31.19:4841 , piece_id=SL2XF2NPCJ4REJ7FMH4KQ5R7URDBBXR2IC4GKWEB5XWUHXWOVC7A, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.231935166 (604h27m37.103018201s) [noise=true , address=45.91.116.8:4046 , piece_id=FJRJS3ZJTEXIPK4TKCTLVSCE6RYFFKY64KWLYHK6IW52E7OM7NQQ, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.232619950 (604h27m37.103702995s) [noise=true , address=45.91.116.9:4264 , piece_id=XEJJGSR3H2PEXM7YWNCUVZYXGWQ63FL7MBSQG3FSN7IWRLAWWUIA, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.233080628 (604h27m37.104163663s) [noise=true , address=145.79.227.4:4186 , piece_id=CUF5JYTXABJ5VOWPWCUCSZXBDHLY5EM2JGES4HRXDCHVY2IAY3TQ, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:38.941787109 (604h27m38.812870144s) [upload=529794]: Next returned nil segment Dec 30 19:56:38.941854097 (604h27m38.812937162s) [upload=529794]: Waiting for error group managing segments... Dec 30 19:56:39.223299549 (604h27m39.094382584s) [noise=true , address=145.79.227.4:4186 , piece_id=CUF5JYTXABJ5VOWPWCUCSZXBDHLY5EM2JGES4HRXDCHVY2IAY3TQ, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.229871974 (604h27m39.100955009s) [noise=true , address=145.79.227.38:4979 , piece_id=K5YWQA3JXONYZGPNBW27ZCWAAIXDYXTHCYXXVHI4QATMTXHNXGHQ, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.233639237 (604h27m39.104722282s) [noise=true , address=145.79.227.45:4229 , piece_id=EXTZJT63CIG6MYSAOTQ6FS24CWQ26P5SABM2XCGLGO5S6KQYFSLA, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.234939713 (604h27m39.106022748s) [noise=true , address=145.79.227.8:4330 , piece_id=5VDJ3GBK43DHSGSOAS3P74PHDQUHCRTX6ADXXYNKKYFBTSLXC6UA, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.241257905 (604h27m39.11234094s) [noise=true , address=145.79.227.39:4015 , piece_id=AMGJEADOFORWWDEB5YWQKWSBER5F4G4P2XY3VOW66LCFVVYPZ5HQ, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.249599698 (604h27m39.120682733s) [noise=true , address=145.79.227.18:4007 , piece_id=KPR5I4W6SHDPL5H6ICSQYAV6IMJ6SBT6HBLVZPCOWNBJB24L2YJA, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.267283862 (604h27m39.138366897s) [noise=true , address=145.79.227.17:4970 , piece_id=VN3JRKKNAGSJK6QK3LJRCM553O4POX6K7CL2OZPT33WVC2W3Y3UA, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.316366802 (604h27m39.187449837s) [noise=true , address=145.79.227.32:4835 , piece_id=BGKMICBP2KWG6PO4AUSXNN5QKKILJ3THBENIXREYQCIWVVAIRQWQ, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.453863376 (604h27m39.324946411s) [noise=true , address=77.90.31.17:4770 , piece_id=PXB64XW2RPG3Y7EJFF4VSH2SVHBLGNCQSROV424VVVH7E7XEOIVA, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.489946233 (604h27m39.361029268s) [noise=true , address=77.90.31.15:4697 , piece_id=E2WQ2IE4XEHNLT5ZOI36XBVWKUTT2XKQCMMCJHGISGW5PM23OSPA, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.492675480 (604h27m39.363758525s) [noise=true , address=77.90.31.14:4664 , piece_id=7OTARUUJ5SMUPWNWGJU7UPZ4BH6EXUNBOLHVDRQKCPS26VRVNQTQ, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.580589390 (604h27m39.451672435s) [noise=true , address=45.91.116.13:4155 , piece_id=CKYZYLMMWE5BA25VSXZY7XPROPFU6NPKXQPDJ7FMUASFNSYLO2ZA, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.580711223 (604h27m39.451794268s) [noise=true , address=45.91.116.9:4264 , piece_id=XEJJGSR3H2PEXM7YWNCUVZYXGWQ63FL7MBSQG3FSN7IWRLAWWUIA, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.580734668 (604h27m39.451817713s) [seg_pos={0 0}, upload=529794]: Segment reached optimal threshold of 13 pieces. Dec 30 19:56:39.580946010 (604h27m39.452029055s) [noise=true , address=45.91.116.5:4298 , piece_id=QZULXYQVEFQBIDBZBDLT7Z67X5LK4L4JNNXYG2UYKWMIHAJDEPDQ, seg_pos={0 0}, upload=529794]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 30 19:56:39.580964675 (604h27m39.45204771s) [noise=true , address=45.91.116.5:4298 , piece_id=QZULXYQVEFQBIDBZBDLT7Z67X5LK4L4JNNXYG2UYKWMIHAJDEPDQ, seg_pos={0 0}, upload=529794]: Long tail cancellation due to optimal threshold Dec 30 19:56:39.581009039 (604h27m39.452092095s) [noise=true , address=45.91.116.8:4046 , piece_id=FJRJS3ZJTEXIPK4TKCTLVSCE6RYFFKY64KWLYHK6IW52E7OM7NQQ, seg_pos={0 0}, upload=529794]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 30 19:56:39.581019841 (604h27m39.452102896s) [noise=true , address=45.91.116.8:4046 , piece_id=FJRJS3ZJTEXIPK4TKCTLVSCE6RYFFKY64KWLYHK6IW52E7OM7NQQ, seg_pos={0 0}, upload=529794]: Long tail cancellation due to optimal threshold Dec 30 19:56:39.581069174 (604h27m39.452152209s) [noise=true , address=77.90.31.19:4841 , piece_id=SL2XF2NPCJ4REJ7FMH4KQ5R7URDBBXR2IC4GKWEB5XWUHXWOVC7A, seg_pos={0 0}, upload=529794]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Dec 30 19:56:39.581076298 (604h27m39.452159333s) [noise=true , address=77.90.31.19:4841 , piece_id=SL2XF2NPCJ4REJ7FMH4KQ5R7URDBBXR2IC4GKWEB5XWUHXWOVC7A, seg_pos={0 0}, upload=529794]: Long tail cancellation due to optimal threshold Dec 30 19:56:39.581097528 (604h27m39.452180563s) [seg_pos={0 0}, upload=529794]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 19:56:39.581197789 (604h27m39.452280824s) [upload=529794]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:56:39.581204532 (604h27m39.452287567s) [upload=529794]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:56:39.872427473 (604h27m39.743510508s) [upload=529794]: Done uploading segments... Dec 30 19:56:39.872446579 (604h27m39.743529624s) [upload=529794]: Upload finished. err: <nil> Duration: 29.173528345s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [....X..X.....................X......................X....XX.] duration: 22.970511942s trace: https://jaeger.dev.storj.io/trace/840262455c60921 FAIL (2025-12-30 19:57:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ, Piece ID: MNIZD24L22A677AVYFP2JTJ76YUO73X3YOJ3GBOBTEUTXLCORIEQ) context deadline exceeded; piecestore: (Node ID: 1WpVJa7SXkANKParDWYrC1cN8aLHTkUSPrRJGKQ6fvA5oDEkoa, Piece ID: LLKCRW4FFBCZJYEONAPY2PBBIUA4PHJ6VWGGARB6DEDO7KONFSXA) context deadline exceeded; piecestore: (Node ID: 1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, Piece ID: FD6B7RMU7XCL6ZI4FSG77JBTFGJNORR4CYN4RBRXKH7UGYQVCHUA) context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.40:4075 , piece_id=DZXOXPZ73ZEAWIDRN7AASB4EO4INTJHOVJ342HD5O4CYONEVAICQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.992183721 (604h27m6.863266766s) [noise=true , address=5.231.73.5:4019 , piece_id=DLKN6PTMXQGLSGHSCXJLRCTJ3QKAVXHXTE33R66HPMBMFMNZWQDA, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.992249356 (604h27m6.863332391s) [noise=true , address=77.90.31.10:4521 , piece_id=LLKCRW4FFBCZJYEONAPY2PBBIUA4PHJ6VWGGARB6DEDO7KONFSXA, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.992522767 (604h27m6.863605802s) [noise=true , address=145.79.227.6:4281 , piece_id=IJFEH45LLCFVNT3VURMQ2TWKDTKAPDLIA6DRP4ECGA6M36IHURVQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.992532625 (604h27m6.86361566s) [noise=true , address=145.79.227.39:4016 , piece_id=UK7PLDGLWUXXFBETE2T3LFBHIYHLO6Y4422XYN36AOXHCI3LBFFQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.992817888 (604h27m6.863900933s) [noise=true , address=145.79.227.22:4537 , piece_id=MKOEP4MXR2SJX4EDOIY7UDJ4G25Y5FAJ4AVAZ6KBFYKG45EDJ4TQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.993072103 (604h27m6.864155138s) [noise=true , address=77.90.31.12:4601 , piece_id=Z2TMMXC2Q3O5E4WWZ6PMZGWOBNUPSUMNKUPZCFKJOD6TV2QVDXNQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.993346675 (604h27m6.86442971s) [noise=true , address=145.79.227.49:4344 , piece_id=5DR3HZYF36VOGJ53WOZSB454C7MOWBOYE7U6IGGVTMTG4GO5DA5A, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.993630466 (604h27m6.864713501s) [noise=true , address=77.90.31.21:4891 , piece_id=KPW3XZXEW3BVHIITM4HKYFU5BEF4SVOOYRFWZENES7F7MBD2SX2A, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.993927371 (604h27m6.865010416s) [noise=true , address=145.79.227.30:4775 , piece_id=OYD6XLL2744IECKVTY5ECJUHCEWS4O7LALWZRRCNYBGER4LYVJVQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.994315810 (604h27m6.865398846s) [noise=true , address=77.90.31.17:4790 , piece_id=FD6B7RMU7XCL6ZI4FSG77JBTFGJNORR4CYN4RBRXKH7UGYQVCHUA, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.994706845 (604h27m6.86578988s) [noise=true , address=145.79.227.34:4927 , piece_id=MQZMI72BZ6JXVA3W3YHKSAYMNTPQWEPYX57TETX2QOX4BHWKZ2WA, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.995103571 (604h27m6.866186606s) [noise=true , address=45.91.116.13:4176 , piece_id=MNIZD24L22A677AVYFP2JTJ76YUO73X3YOJ3GBOBTEUTXLCORIEQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.995463917 (604h27m6.866546952s) [noise=true , address=64.49.13.139:4475 , piece_id=7JNXM7PAXURJP2X774PZ56DOFHKXMSUDJGYOXKURIRGZFONRVVOQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.996040635 (604h27m6.86712367s) [noise=true , address=145.79.227.21:4480 , piece_id=L62OLPCCYDK5EDOPPTKEITJOIXYQJZKROEP3YES6EQSHPUR7B2BA, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:08.448203770 (604h27m8.319286805s) [upload=529788]: Next returned nil segment Dec 30 19:56:08.448226784 (604h27m8.319309819s) [upload=529788]: Waiting for error group managing segments... Dec 30 19:56:08.856597951 (604h27m8.727680986s) [noise=true , address=145.79.227.22:4537 , piece_id=MKOEP4MXR2SJX4EDOIY7UDJ4G25Y5FAJ4AVAZ6KBFYKG45EDJ4TQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:08.972642035 (604h27m8.84372508s) [noise=true , address=64.49.13.139:4475 , piece_id=7JNXM7PAXURJP2X774PZ56DOFHKXMSUDJGYOXKURIRGZFONRVVOQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:08.982556104 (604h27m8.853639149s) [noise=true , address=5.231.73.5:4019 , piece_id=DLKN6PTMXQGLSGHSCXJLRCTJ3QKAVXHXTE33R66HPMBMFMNZWQDA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.017622445 (604h27m8.88870549s) [noise=true , address=77.90.31.10:4521 , piece_id=LLKCRW4FFBCZJYEONAPY2PBBIUA4PHJ6VWGGARB6DEDO7KONFSXA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.096193105 (604h27m8.96727614s) [noise=true , address=77.90.31.12:4601 , piece_id=Z2TMMXC2Q3O5E4WWZ6PMZGWOBNUPSUMNKUPZCFKJOD6TV2QVDXNQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.120549052 (604h27m8.991632097s) [noise=true , address=77.90.31.13:4655 , piece_id=ALKFNT4I4ZBK24CFWVVFHJNH4FKT6X4QO5LU5IQXLDSBNGU5FY6Q, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.121572410 (604h27m8.992655455s) [noise=true , address=77.90.31.21:4891 , piece_id=KPW3XZXEW3BVHIITM4HKYFU5BEF4SVOOYRFWZENES7F7MBD2SX2A, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.171218263 (604h27m9.042301298s) [noise=true , address=45.91.116.13:4176 , piece_id=MNIZD24L22A677AVYFP2JTJ76YUO73X3YOJ3GBOBTEUTXLCORIEQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.480613004 (604h27m9.351696049s) [noise=true , address=77.90.31.17:4790 , piece_id=FD6B7RMU7XCL6ZI4FSG77JBTFGJNORR4CYN4RBRXKH7UGYQVCHUA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.779927943 (604h27m9.651010978s) [noise=true , address=145.79.227.30:4775 , piece_id=OYD6XLL2744IECKVTY5ECJUHCEWS4O7LALWZRRCNYBGER4LYVJVQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.809206594 (604h27m9.680289649s) [noise=true , address=145.79.227.49:4344 , piece_id=5DR3HZYF36VOGJ53WOZSB454C7MOWBOYE7U6IGGVTMTG4GO5DA5A, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.882970882 (604h27m9.754053917s) [noise=true , address=145.79.227.34:4927 , piece_id=MQZMI72BZ6JXVA3W3YHKSAYMNTPQWEPYX57TETX2QOX4BHWKZ2WA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.894950363 (604h27m9.766033398s) [noise=true , address=145.79.227.44:4198 , piece_id=HTTP7BJLLNNP7BKCKC43VRRTJWESVCNS6HDMGEPLRN5GMVQHHQKA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.895290701 (604h27m9.766373736s) [noise=true , address=145.79.227.43:4189 , piece_id=SAUQY62J6PECMGHJQTSXRD6NNFWFE6A5AHVQYA3UW3DHLZUXFPIQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.895298405 (604h27m9.766381451s) [seg_pos={0 0}, upload=529788]: Segment reached optimal threshold of 14 pieces. Dec 30 19:56:09.895436178 (604h27m9.766519213s) [noise=true , address=145.79.227.21:4480 , piece_id=L62OLPCCYDK5EDOPPTKEITJOIXYQJZKROEP3YES6EQSHPUR7B2BA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qAtwKs88zrSjvRR9fvez9J9HnRBF2Ax7APfCq9BLHSFTE24a6): protocol: write tcp 188.166.225.170:59714->145.79.227.21:4480: use of closed network connection; write tcp 188.166.225.170:59714->145.79.227.21:4480: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59714->145.79.227.21:4480: use of closed network connection Dec 30 19:56:09.895442039 (604h27m9.766525074s) [noise=true , address=145.79.227.21:4480 , piece_id=L62OLPCCYDK5EDOPPTKEITJOIXYQJZKROEP3YES6EQSHPUR7B2BA, seg_pos={0 0}, upload=529788]: Long tail cancellation due to optimal threshold Dec 30 19:56:09.895651538 (604h27m9.766734573s) [noise=true , address=145.79.227.39:4016 , piece_id=UK7PLDGLWUXXFBETE2T3LFBHIYHLO6Y4422XYN36AOXHCI3LBFFQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jC98jEWn6K8jFHPA9knJeHAkuNEmcS7ThxUvbySDfbq6FmKVR): protocol: write tcp 188.166.225.170:39076->145.79.227.39:4016: use of closed network connection; write tcp 188.166.225.170:39076->145.79.227.39:4016: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39076->145.79.227.39:4016: use of closed network connection Dec 30 19:56:09.895656938 (604h27m9.766739973s) [noise=true , address=145.79.227.39:4016 , piece_id=UK7PLDGLWUXXFBETE2T3LFBHIYHLO6Y4422XYN36AOXHCI3LBFFQ, seg_pos={0 0}, upload=529788]: Long tail cancellation due to optimal threshold Dec 30 19:56:09.895753382 (604h27m9.766836417s) [noise=true , address=145.79.227.40:4075 , piece_id=DZXOXPZ73ZEAWIDRN7AASB4EO4INTJHOVJ342HD5O4CYONEVAICQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 19:56:09.895758010 (604h27m9.766841045s) [noise=true , address=145.79.227.40:4075 , piece_id=DZXOXPZ73ZEAWIDRN7AASB4EO4INTJHOVJ342HD5O4CYONEVAICQ, seg_pos={0 0}, upload=529788]: Long tail cancellation due to optimal threshold Dec 30 19:56:09.895833083 (604h27m9.766916118s) [noise=true , address=145.79.227.6:4281 , piece_id=IJFEH45LLCFVNT3VURMQ2TWKDTKAPDLIA6DRP4ECGA6M36IHURVQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hDXeAm4tomp1gAkBNjM8x77V76yiruYQmPQ3BZEYPLpJzigo2): protocol: expected piece hash; context canceled; EOF Dec 30 19:56:09.895837542 (604h27m9.766920577s) [noise=true , address=145.79.227.6:4281 , piece_id=IJFEH45LLCFVNT3VURMQ2TWKDTKAPDLIA6DRP4ECGA6M36IHURVQ, seg_pos={0 0}, upload=529788]: Long tail cancellation due to optimal threshold Dec 30 19:56:09.895850146 (604h27m9.766933191s) [seg_pos={0 0}, upload=529788]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 19:56:09.895926181 (604h27m9.767009216s) [upload=529788]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:56:09.895933535 (604h27m9.76701657s) [upload=529788]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:56:10.230726588 (604h27m10.101809633s) [upload=529788]: Done uploading segments... Dec 30 19:56:10.230750393 (604h27m10.101833439s) [upload=529788]: Upload finished. err: <nil> Duration: 22.970527982s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.X..XX...X..X.....XX.X.....X.....X.................X...X.X.X] duration: 30.002880644s trace: https://jaeger.dev.storj.io/trace/796612bf4f96ea62 FAIL (2025-12-30 19:57:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: 7OTARUUJ5SMUPWNWGJU7UPZ4BH6EXUNBOLHVDRQKCPS26VRVNQTQ) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: E2WQ2IE4XEHNLT5ZOI36XBVWKUTT2XKQCMMCJHGISGW5PM23OSPA) context deadline exceeded ======LOGS====== Dec 30 19:56:36.978167026 (604h27m36.849250061s) [upload=529794]: Starting upload Dec 30 19:56:36.978278318 (604h27m36.849361373s) [upload=529794]: Done starting upload Dec 30 19:56:36.978339805 (604h27m36.84942284s) [upload=529794]: Uploading segments... Dec 30 19:56:36.978599078 (604h27m36.849682113s) [upload=529794]: Got next segment. Inline: false Dec 30 19:56:36.978635257 (604h27m36.849718292s) [upload=529794]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 19:56:36.978640226 (604h27m36.849723261s) [upload=529794]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 19:56:37.224970965 (604h27m37.096054s) [seg_pos={0 0}, upload=529794]: Begin upload segment... Dec 30 19:56:37.225024648 (604h27m37.096107683s) [seg_pos={0 0}, upload=529794]: Done begin upload segment. Dec 30 19:56:37.225055818 (604h27m37.096138853s) [noise=true , address=145.79.227.45:4229 , piece_id=EXTZJT63CIG6MYSAOTQ6FS24CWQ26P5SABM2XCGLGO5S6KQYFSLA, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.225073160 (604h27m37.096156205s) [noise=true , address=45.91.116.5:4298 , piece_id=QZULXYQVEFQBIDBZBDLT7Z67X5LK4L4JNNXYG2UYKWMIHAJDEPDQ, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.225903902 (604h27m37.096986967s) [noise=true , address=145.79.227.17:4970 , piece_id=VN3JRKKNAGSJK6QK3LJRCM553O4POX6K7CL2OZPT33WVC2W3Y3UA, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.226564250 (604h27m37.097647295s) [noise=true , address=145.79.227.18:4007 , piece_id=KPR5I4W6SHDPL5H6ICSQYAV6IMJ6SBT6HBLVZPCOWNBJB24L2YJA, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.227046768 (604h27m37.098129804s) [noise=true , address=145.79.227.39:4015 , piece_id=AMGJEADOFORWWDEB5YWQKWSBER5F4G4P2XY3VOW66LCFVVYPZ5HQ, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.227690133 (604h27m37.098773168s) [noise=true , address=77.90.31.17:4770 , piece_id=PXB64XW2RPG3Y7EJFF4VSH2SVHBLGNCQSROV424VVVH7E7XEOIVA, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.228146492 (604h27m37.099229547s) [noise=true , address=145.79.227.32:4835 , piece_id=BGKMICBP2KWG6PO4AUSXNN5QKKILJ3THBENIXREYQCIWVVAIRQWQ, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.228545893 (604h27m37.099628928s) [noise=true , address=45.91.116.13:4155 , piece_id=CKYZYLMMWE5BA25VSXZY7XPROPFU6NPKXQPDJ7FMUASFNSYLO2ZA, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.228926797 (604h27m37.100009832s) [noise=true , address=77.90.31.15:4697 , piece_id=E2WQ2IE4XEHNLT5ZOI36XBVWKUTT2XKQCMMCJHGISGW5PM23OSPA, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.229268368 (604h27m37.100351403s) [noise=true , address=77.90.31.14:4664 , piece_id=7OTARUUJ5SMUPWNWGJU7UPZ4BH6EXUNBOLHVDRQKCPS26VRVNQTQ, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.229587335 (604h27m37.10067037s) [noise=true , address=145.79.227.8:4330 , piece_id=5VDJ3GBK43DHSGSOAS3P74PHDQUHCRTX6ADXXYNKKYFBTSLXC6UA, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.229993659 (604h27m37.101076694s) [noise=true , address=145.79.227.38:4979 , piece_id=K5YWQA3JXONYZGPNBW27ZCWAAIXDYXTHCYXXVHI4QATMTXHNXGHQ, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.230365578 (604h27m37.101448613s) [noise=true , address=77.90.31.19:4841 , piece_id=SL2XF2NPCJ4REJ7FMH4KQ5R7URDBBXR2IC4GKWEB5XWUHXWOVC7A, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.231935166 (604h27m37.103018201s) [noise=true , address=45.91.116.8:4046 , piece_id=FJRJS3ZJTEXIPK4TKCTLVSCE6RYFFKY64KWLYHK6IW52E7OM7NQQ, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.232619950 (604h27m37.103702995s) [noise=true , address=45.91.116.9:4264 , piece_id=XEJJGSR3H2PEXM7YWNCUVZYXGWQ63FL7MBSQG3FSN7IWRLAWWUIA, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:37.233080628 (604h27m37.104163663s) [noise=true , address=145.79.227.4:4186 , piece_id=CUF5JYTXABJ5VOWPWCUCSZXBDHLY5EM2JGES4HRXDCHVY2IAY3TQ, seg_pos={0 0}, upload=529794]: Uploading piece... Dec 30 19:56:38.941787109 (604h27m38.812870144s) [upload=529794]: Next returned nil segment Dec 30 19:56:38.941854097 (604h27m38.812937162s) [upload=529794]: Waiting for error group managing segments... Dec 30 19:56:39.223299549 (604h27m39.094382584s) [noise=true , address=145.79.227.4:4186 , piece_id=CUF5JYTXABJ5VOWPWCUCSZXBDHLY5EM2JGES4HRXDCHVY2IAY3TQ, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.229871974 (604h27m39.100955009s) [noise=true , address=145.79.227.38:4979 , piece_id=K5YWQA3JXONYZGPNBW27ZCWAAIXDYXTHCYXXVHI4QATMTXHNXGHQ, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.233639237 (604h27m39.104722282s) [noise=true , address=145.79.227.45:4229 , piece_id=EXTZJT63CIG6MYSAOTQ6FS24CWQ26P5SABM2XCGLGO5S6KQYFSLA, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.234939713 (604h27m39.106022748s) [noise=true , address=145.79.227.8:4330 , piece_id=5VDJ3GBK43DHSGSOAS3P74PHDQUHCRTX6ADXXYNKKYFBTSLXC6UA, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.241257905 (604h27m39.11234094s) [noise=true , address=145.79.227.39:4015 , piece_id=AMGJEADOFORWWDEB5YWQKWSBER5F4G4P2XY3VOW66LCFVVYPZ5HQ, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.249599698 (604h27m39.120682733s) [noise=true , address=145.79.227.18:4007 , piece_id=KPR5I4W6SHDPL5H6ICSQYAV6IMJ6SBT6HBLVZPCOWNBJB24L2YJA, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.267283862 (604h27m39.138366897s) [noise=true , address=145.79.227.17:4970 , piece_id=VN3JRKKNAGSJK6QK3LJRCM553O4POX6K7CL2OZPT33WVC2W3Y3UA, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.316366802 (604h27m39.187449837s) [noise=true , address=145.79.227.32:4835 , piece_id=BGKMICBP2KWG6PO4AUSXNN5QKKILJ3THBENIXREYQCIWVVAIRQWQ, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.453863376 (604h27m39.324946411s) [noise=true , address=77.90.31.17:4770 , piece_id=PXB64XW2RPG3Y7EJFF4VSH2SVHBLGNCQSROV424VVVH7E7XEOIVA, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.489946233 (604h27m39.361029268s) [noise=true , address=77.90.31.15:4697 , piece_id=E2WQ2IE4XEHNLT5ZOI36XBVWKUTT2XKQCMMCJHGISGW5PM23OSPA, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.492675480 (604h27m39.363758525s) [noise=true , address=77.90.31.14:4664 , piece_id=7OTARUUJ5SMUPWNWGJU7UPZ4BH6EXUNBOLHVDRQKCPS26VRVNQTQ, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.580589390 (604h27m39.451672435s) [noise=true , address=45.91.116.13:4155 , piece_id=CKYZYLMMWE5BA25VSXZY7XPROPFU6NPKXQPDJ7FMUASFNSYLO2ZA, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.580711223 (604h27m39.451794268s) [noise=true , address=45.91.116.9:4264 , piece_id=XEJJGSR3H2PEXM7YWNCUVZYXGWQ63FL7MBSQG3FSN7IWRLAWWUIA, seg_pos={0 0}, upload=529794]: Done uploading piece. err: <nil> Dec 30 19:56:39.580734668 (604h27m39.451817713s) [seg_pos={0 0}, upload=529794]: Segment reached optimal threshold of 13 pieces. Dec 30 19:56:39.580946010 (604h27m39.452029055s) [noise=true , address=45.91.116.5:4298 , piece_id=QZULXYQVEFQBIDBZBDLT7Z67X5LK4L4JNNXYG2UYKWMIHAJDEPDQ, seg_pos={0 0}, upload=529794]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 30 19:56:39.580964675 (604h27m39.45204771s) [noise=true , address=45.91.116.5:4298 , piece_id=QZULXYQVEFQBIDBZBDLT7Z67X5LK4L4JNNXYG2UYKWMIHAJDEPDQ, seg_pos={0 0}, upload=529794]: Long tail cancellation due to optimal threshold Dec 30 19:56:39.581009039 (604h27m39.452092095s) [noise=true , address=45.91.116.8:4046 , piece_id=FJRJS3ZJTEXIPK4TKCTLVSCE6RYFFKY64KWLYHK6IW52E7OM7NQQ, seg_pos={0 0}, upload=529794]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 30 19:56:39.581019841 (604h27m39.452102896s) [noise=true , address=45.91.116.8:4046 , piece_id=FJRJS3ZJTEXIPK4TKCTLVSCE6RYFFKY64KWLYHK6IW52E7OM7NQQ, seg_pos={0 0}, upload=529794]: Long tail cancellation due to optimal threshold Dec 30 19:56:39.581069174 (604h27m39.452152209s) [noise=true , address=77.90.31.19:4841 , piece_id=SL2XF2NPCJ4REJ7FMH4KQ5R7URDBBXR2IC4GKWEB5XWUHXWOVC7A, seg_pos={0 0}, upload=529794]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF Dec 30 19:56:39.581076298 (604h27m39.452159333s) [noise=true , address=77.90.31.19:4841 , piece_id=SL2XF2NPCJ4REJ7FMH4KQ5R7URDBBXR2IC4GKWEB5XWUHXWOVC7A, seg_pos={0 0}, upload=529794]: Long tail cancellation due to optimal threshold Dec 30 19:56:39.581097528 (604h27m39.452180563s) [seg_pos={0 0}, upload=529794]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 19:56:39.581197789 (604h27m39.452280824s) [upload=529794]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:56:39.581204532 (604h27m39.452287567s) [upload=529794]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:56:39.872427473 (604h27m39.743510508s) [upload=529794]: Done uploading segments... Dec 30 19:56:39.872446579 (604h27m39.743529624s) [upload=529794]: Upload finished. err: <nil> Duration: 30.002914128s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....X..X.....................X......................X....XX] duration: 1m0.002687312s trace: https://jaeger.dev.storj.io/trace/840262455c60921 FAIL (2025-12-30 19:57:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ejsmBd4tAVwTHjamPYbryQb5x2vDnk9pBuX3CBF9WYFhTrFVJ, Piece ID: MNIZD24L22A677AVYFP2JTJ76YUO73X3YOJ3GBOBTEUTXLCORIEQ) context deadline exceeded; piecestore: (Node ID: 1WpVJa7SXkANKParDWYrC1cN8aLHTkUSPrRJGKQ6fvA5oDEkoa, Piece ID: LLKCRW4FFBCZJYEONAPY2PBBIUA4PHJ6VWGGARB6DEDO7KONFSXA) context deadline exceeded; piecestore: (Node ID: 1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, Piece ID: FD6B7RMU7XCL6ZI4FSG77JBTFGJNORR4CYN4RBRXKH7UGYQVCHUA) context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.40:4075 , piece_id=DZXOXPZ73ZEAWIDRN7AASB4EO4INTJHOVJ342HD5O4CYONEVAICQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.992183721 (604h27m6.863266766s) [noise=true , address=5.231.73.5:4019 , piece_id=DLKN6PTMXQGLSGHSCXJLRCTJ3QKAVXHXTE33R66HPMBMFMNZWQDA, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.992249356 (604h27m6.863332391s) [noise=true , address=77.90.31.10:4521 , piece_id=LLKCRW4FFBCZJYEONAPY2PBBIUA4PHJ6VWGGARB6DEDO7KONFSXA, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.992522767 (604h27m6.863605802s) [noise=true , address=145.79.227.6:4281 , piece_id=IJFEH45LLCFVNT3VURMQ2TWKDTKAPDLIA6DRP4ECGA6M36IHURVQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.992532625 (604h27m6.86361566s) [noise=true , address=145.79.227.39:4016 , piece_id=UK7PLDGLWUXXFBETE2T3LFBHIYHLO6Y4422XYN36AOXHCI3LBFFQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.992817888 (604h27m6.863900933s) [noise=true , address=145.79.227.22:4537 , piece_id=MKOEP4MXR2SJX4EDOIY7UDJ4G25Y5FAJ4AVAZ6KBFYKG45EDJ4TQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.993072103 (604h27m6.864155138s) [noise=true , address=77.90.31.12:4601 , piece_id=Z2TMMXC2Q3O5E4WWZ6PMZGWOBNUPSUMNKUPZCFKJOD6TV2QVDXNQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.993346675 (604h27m6.86442971s) [noise=true , address=145.79.227.49:4344 , piece_id=5DR3HZYF36VOGJ53WOZSB454C7MOWBOYE7U6IGGVTMTG4GO5DA5A, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.993630466 (604h27m6.864713501s) [noise=true , address=77.90.31.21:4891 , piece_id=KPW3XZXEW3BVHIITM4HKYFU5BEF4SVOOYRFWZENES7F7MBD2SX2A, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.993927371 (604h27m6.865010416s) [noise=true , address=145.79.227.30:4775 , piece_id=OYD6XLL2744IECKVTY5ECJUHCEWS4O7LALWZRRCNYBGER4LYVJVQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.994315810 (604h27m6.865398846s) [noise=true , address=77.90.31.17:4790 , piece_id=FD6B7RMU7XCL6ZI4FSG77JBTFGJNORR4CYN4RBRXKH7UGYQVCHUA, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.994706845 (604h27m6.86578988s) [noise=true , address=145.79.227.34:4927 , piece_id=MQZMI72BZ6JXVA3W3YHKSAYMNTPQWEPYX57TETX2QOX4BHWKZ2WA, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.995103571 (604h27m6.866186606s) [noise=true , address=45.91.116.13:4176 , piece_id=MNIZD24L22A677AVYFP2JTJ76YUO73X3YOJ3GBOBTEUTXLCORIEQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.995463917 (604h27m6.866546952s) [noise=true , address=64.49.13.139:4475 , piece_id=7JNXM7PAXURJP2X774PZ56DOFHKXMSUDJGYOXKURIRGZFONRVVOQ, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:06.996040635 (604h27m6.86712367s) [noise=true , address=145.79.227.21:4480 , piece_id=L62OLPCCYDK5EDOPPTKEITJOIXYQJZKROEP3YES6EQSHPUR7B2BA, seg_pos={0 0}, upload=529788]: Uploading piece... Dec 30 19:56:08.448203770 (604h27m8.319286805s) [upload=529788]: Next returned nil segment Dec 30 19:56:08.448226784 (604h27m8.319309819s) [upload=529788]: Waiting for error group managing segments... Dec 30 19:56:08.856597951 (604h27m8.727680986s) [noise=true , address=145.79.227.22:4537 , piece_id=MKOEP4MXR2SJX4EDOIY7UDJ4G25Y5FAJ4AVAZ6KBFYKG45EDJ4TQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:08.972642035 (604h27m8.84372508s) [noise=true , address=64.49.13.139:4475 , piece_id=7JNXM7PAXURJP2X774PZ56DOFHKXMSUDJGYOXKURIRGZFONRVVOQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:08.982556104 (604h27m8.853639149s) [noise=true , address=5.231.73.5:4019 , piece_id=DLKN6PTMXQGLSGHSCXJLRCTJ3QKAVXHXTE33R66HPMBMFMNZWQDA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.017622445 (604h27m8.88870549s) [noise=true , address=77.90.31.10:4521 , piece_id=LLKCRW4FFBCZJYEONAPY2PBBIUA4PHJ6VWGGARB6DEDO7KONFSXA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.096193105 (604h27m8.96727614s) [noise=true , address=77.90.31.12:4601 , piece_id=Z2TMMXC2Q3O5E4WWZ6PMZGWOBNUPSUMNKUPZCFKJOD6TV2QVDXNQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.120549052 (604h27m8.991632097s) [noise=true , address=77.90.31.13:4655 , piece_id=ALKFNT4I4ZBK24CFWVVFHJNH4FKT6X4QO5LU5IQXLDSBNGU5FY6Q, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.121572410 (604h27m8.992655455s) [noise=true , address=77.90.31.21:4891 , piece_id=KPW3XZXEW3BVHIITM4HKYFU5BEF4SVOOYRFWZENES7F7MBD2SX2A, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.171218263 (604h27m9.042301298s) [noise=true , address=45.91.116.13:4176 , piece_id=MNIZD24L22A677AVYFP2JTJ76YUO73X3YOJ3GBOBTEUTXLCORIEQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.480613004 (604h27m9.351696049s) [noise=true , address=77.90.31.17:4790 , piece_id=FD6B7RMU7XCL6ZI4FSG77JBTFGJNORR4CYN4RBRXKH7UGYQVCHUA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.779927943 (604h27m9.651010978s) [noise=true , address=145.79.227.30:4775 , piece_id=OYD6XLL2744IECKVTY5ECJUHCEWS4O7LALWZRRCNYBGER4LYVJVQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.809206594 (604h27m9.680289649s) [noise=true , address=145.79.227.49:4344 , piece_id=5DR3HZYF36VOGJ53WOZSB454C7MOWBOYE7U6IGGVTMTG4GO5DA5A, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.882970882 (604h27m9.754053917s) [noise=true , address=145.79.227.34:4927 , piece_id=MQZMI72BZ6JXVA3W3YHKSAYMNTPQWEPYX57TETX2QOX4BHWKZ2WA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.894950363 (604h27m9.766033398s) [noise=true , address=145.79.227.44:4198 , piece_id=HTTP7BJLLNNP7BKCKC43VRRTJWESVCNS6HDMGEPLRN5GMVQHHQKA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.895290701 (604h27m9.766373736s) [noise=true , address=145.79.227.43:4189 , piece_id=SAUQY62J6PECMGHJQTSXRD6NNFWFE6A5AHVQYA3UW3DHLZUXFPIQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: <nil> Dec 30 19:56:09.895298405 (604h27m9.766381451s) [seg_pos={0 0}, upload=529788]: Segment reached optimal threshold of 14 pieces. Dec 30 19:56:09.895436178 (604h27m9.766519213s) [noise=true , address=145.79.227.21:4480 , piece_id=L62OLPCCYDK5EDOPPTKEITJOIXYQJZKROEP3YES6EQSHPUR7B2BA, seg_pos={0 0}, upload=529788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qAtwKs88zrSjvRR9fvez9J9HnRBF2Ax7APfCq9BLHSFTE24a6): protocol: write tcp 188.166.225.170:59714->145.79.227.21:4480: use of closed network connection; write tcp 188.166.225.170:59714->145.79.227.21:4480: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59714->145.79.227.21:4480: use of closed network connection Dec 30 19:56:09.895442039 (604h27m9.766525074s) [noise=true , address=145.79.227.21:4480 , piece_id=L62OLPCCYDK5EDOPPTKEITJOIXYQJZKROEP3YES6EQSHPUR7B2BA, seg_pos={0 0}, upload=529788]: Long tail cancellation due to optimal threshold Dec 30 19:56:09.895651538 (604h27m9.766734573s) [noise=true , address=145.79.227.39:4016 , piece_id=UK7PLDGLWUXXFBETE2T3LFBHIYHLO6Y4422XYN36AOXHCI3LBFFQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jC98jEWn6K8jFHPA9knJeHAkuNEmcS7ThxUvbySDfbq6FmKVR): protocol: write tcp 188.166.225.170:39076->145.79.227.39:4016: use of closed network connection; write tcp 188.166.225.170:39076->145.79.227.39:4016: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39076->145.79.227.39:4016: use of closed network connection Dec 30 19:56:09.895656938 (604h27m9.766739973s) [noise=true , address=145.79.227.39:4016 , piece_id=UK7PLDGLWUXXFBETE2T3LFBHIYHLO6Y4422XYN36AOXHCI3LBFFQ, seg_pos={0 0}, upload=529788]: Long tail cancellation due to optimal threshold Dec 30 19:56:09.895753382 (604h27m9.766836417s) [noise=true , address=145.79.227.40:4075 , piece_id=DZXOXPZ73ZEAWIDRN7AASB4EO4INTJHOVJ342HD5O4CYONEVAICQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eTSSYh294r2TyToU22xzVunbLwXwbaVJopvF5ej8RxaDLdbtK): protocol: expected piece hash; context canceled; EOF Dec 30 19:56:09.895758010 (604h27m9.766841045s) [noise=true , address=145.79.227.40:4075 , piece_id=DZXOXPZ73ZEAWIDRN7AASB4EO4INTJHOVJ342HD5O4CYONEVAICQ, seg_pos={0 0}, upload=529788]: Long tail cancellation due to optimal threshold Dec 30 19:56:09.895833083 (604h27m9.766916118s) [noise=true , address=145.79.227.6:4281 , piece_id=IJFEH45LLCFVNT3VURMQ2TWKDTKAPDLIA6DRP4ECGA6M36IHURVQ, seg_pos={0 0}, upload=529788]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hDXeAm4tomp1gAkBNjM8x77V76yiruYQmPQ3BZEYPLpJzigo2): protocol: expected piece hash; context canceled; EOF Dec 30 19:56:09.895837542 (604h27m9.766920577s) [noise=true , address=145.79.227.6:4281 , piece_id=IJFEH45LLCFVNT3VURMQ2TWKDTKAPDLIA6DRP4ECGA6M36IHURVQ, seg_pos={0 0}, upload=529788]: Long tail cancellation due to optimal threshold Dec 30 19:56:09.895850146 (604h27m9.766933191s) [seg_pos={0 0}, upload=529788]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 19:56:09.895926181 (604h27m9.767009216s) [upload=529788]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:56:09.895933535 (604h27m9.76701657s) [upload=529788]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:56:10.230726588 (604h27m10.101809633s) [upload=529788]: Done uploading segments... Dec 30 19:56:10.230750393 (604h27m10.101833439s) [upload=529788]: Upload finished. err: <nil> Duration: 1m0.002697691s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X..X.....................X......................X....X] duration: 1m0.003156993s trace: https://jaeger.dev.storj.io/trace/38224a3744f38e0b FAIL (2025-12-30 19:56:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 365s) [noise=true , address=145.79.227.14:4848 , piece_id=KH3OW5LP7RILI654LPFLH6XDPHABHWPLOZ5XWTIVDLKET7FULMUQ, seg_pos={0 0}, upload=529773]: Uploading piece... Dec 30 19:55:06.767212771 (604h26m6.638295806s) [noise=true , address=145.79.227.23:4553 , piece_id=B62LZF2NQJFU6NBFY5BHOYVYOR35PC2UMLNWYRORYGKXR2LRELXA, seg_pos={0 0}, upload=529773]: Uploading piece... Dec 30 19:55:06.767380941 (604h26m6.638463986s) [noise=true , address=64.49.13.154:4929 , piece_id=5MB5ZIJL4QHWQYZEOBGRTKVSO4F7Z7N4DG4IVY6FTYY3OM4ESHEA, seg_pos={0 0}, upload=529773]: Uploading piece... Dec 30 19:55:06.767725267 (604h26m6.638808312s) [noise=true , address=77.90.31.17:4781 , piece_id=QJOINPKDUTN7SXN4PDMVZXO5IFSBKRZOCUT6PV26KPPYNA3BRVVQ, seg_pos={0 0}, upload=529773]: Uploading piece... Dec 30 19:55:06.768087897 (604h26m6.639170932s) [noise=true , address=45.91.116.4:4944 , piece_id=3AZURY5AJK2PI57ISK4MOGF7EGDMGRA6I3MBKWJPCSB7KK5EJJAQ, seg_pos={0 0}, upload=529773]: Uploading piece... Dec 30 19:55:06.768473170 (604h26m6.639556205s) [noise=true , address=145.79.227.33:4887 , piece_id=OVVPGXJQVMJV5UHHGBRS3MM5PB2L4WDVSMW367D6B6KVKJFTEPLQ, seg_pos={0 0}, upload=529773]: Uploading piece... Dec 30 19:55:06.768922486 (604h26m6.640005521s) [noise=true , address=145.79.227.40:4055 , piece_id=IQ2D66VK2VWUUFUNMMNP4YEECYE7TITDVKWWCJ6I3LWDBHLD4ATQ, seg_pos={0 0}, upload=529773]: Uploading piece... Dec 30 19:55:06.769433880 (604h26m6.640516915s) [noise=true , address=145.79.227.33:4883 , piece_id=PXIEM4ISCK5XV3UBOEE4ILQM2HFRKRWEEHF7FQLKXUBUD5EFGHTQ, seg_pos={0 0}, upload=529773]: Uploading piece... Dec 30 19:55:06.769495728 (604h26m6.640578763s) [noise=true , address=64.49.13.139:4470 , piece_id=HEPLGZIKHDTLSCCQSLW7SKGXE6AB5ECTEN3KOE5PSXDSRBNRAUBQ, seg_pos={0 0}, upload=529773]: Uploading piece... Dec 30 19:55:06.770095189 (604h26m6.641178224s) [noise=true , address=145.79.227.18:4024 , piece_id=WVOWBAGISW2SPCKK3EM3DCTDPDAJE6J3YEOAZLMNKNFZMIW6AQAA, seg_pos={0 0}, upload=529773]: Uploading piece... Dec 30 19:55:06.770569492 (604h26m6.641652537s) [noise=true , address=145.79.227.35:4501 , piece_id=AOTXPNLCU6W27IZXATRTOTE6ZT6UAO53UNFFI6OLGYAGKG4PELPQ, seg_pos={0 0}, upload=529773]: Uploading piece... Dec 30 19:55:06.770925220 (604h26m6.642008255s) [noise=true , address=64.49.13.138:4485 , piece_id=FVQ7F2FXXMVSKWL55PNUMFTL2ECZBARXIVKCOEAOOHNV6QSCOOJQ, seg_pos={0 0}, upload=529773]: Uploading piece... Dec 30 19:55:06.771844099 (604h26m6.642927134s) [noise=true , address=77.90.31.13:4637 , piece_id=6I7C6BUMP5YYT5GWGFOIPN5XMCL27LXPMQEHGS2PK73WQ6TSZ4IA, seg_pos={0 0}, upload=529773]: Uploading piece... Dec 30 19:55:06.772267215 (604h26m6.64335025s) [noise=true , address=45.91.116.14:4212 , piece_id=67CGRBTRLSBNAGL76GGQAGERZ6SLKWNZABNVKNGDNZMQVSD3VSHQ, seg_pos={0 0}, upload=529773]: Uploading piece... Dec 30 19:55:06.772597483 (604h26m6.643680518s) [noise=true , address=145.79.227.34:4922 , piece_id=3LFPTHPY5JUSDGVWZPQY46NPEMDXNOGISRR3H2GXERETTNVOS7FQ, seg_pos={0 0}, upload=529773]: Uploading piece... Dec 30 19:55:08.209678753 (604h26m8.080761788s) [upload=529773]: Next returned nil segment Dec 30 19:55:08.209693691 (604h26m8.080776726s) [upload=529773]: Waiting for error group managing segments... Dec 30 19:55:08.566310670 (604h26m8.437393715s) [noise=true , address=64.49.13.138:4485 , piece_id=FVQ7F2FXXMVSKWL55PNUMFTL2ECZBARXIVKCOEAOOHNV6QSCOOJQ, seg_pos={0 0}, upload=529773]: Done uploading piece. err: <nil> Dec 30 19:55:08.625887953 (604h26m8.496970998s) [noise=true , address=64.49.13.154:4929 , piece_id=5MB5ZIJL4QHWQYZEOBGRTKVSO4F7Z7N4DG4IVY6FTYY3OM4ESHEA, seg_pos={0 0}, upload=529773]: Done uploading piece. err: <nil> Dec 30 19:55:08.629203397 (604h26m8.500286432s) [noise=true , address=64.49.13.139:4470 , piece_id=HEPLGZIKHDTLSCCQSLW7SKGXE6AB5ECTEN3KOE5PSXDSRBNRAUBQ, seg_pos={0 0}, upload=529773]: Done uploading piece. err: <nil> Dec 30 19:55:08.833841547 (604h26m8.704924582s) [noise=true , address=77.90.31.13:4637 , piece_id=6I7C6BUMP5YYT5GWGFOIPN5XMCL27LXPMQEHGS2PK73WQ6TSZ4IA, seg_pos={0 0}, upload=529773]: Done uploading piece. err: <nil> Dec 30 19:55:08.851947794 (604h26m8.723030839s) [noise=true , address=77.90.31.17:4781 , piece_id=QJOINPKDUTN7SXN4PDMVZXO5IFSBKRZOCUT6PV26KPPYNA3BRVVQ, seg_pos={0 0}, upload=529773]: Done uploading piece. err: <nil> Dec 30 19:55:08.985761034 (604h26m8.856844069s) [noise=true , address=45.91.116.14:4212 , piece_id=67CGRBTRLSBNAGL76GGQAGERZ6SLKWNZABNVKNGDNZMQVSD3VSHQ, seg_pos={0 0}, upload=529773]: Done uploading piece. err: <nil> Dec 30 19:55:09.155308948 (604h26m9.026391993s) [noise=true , address=145.79.227.14:4848 , piece_id=KH3OW5LP7RILI654LPFLH6XDPHABHWPLOZ5XWTIVDLKET7FULMUQ, seg_pos={0 0}, upload=529773]: Done uploading piece. err: <nil> Dec 30 19:55:09.606914859 (604h26m9.477997894s) [noise=true , address=77.90.31.19:4871 , piece_id=IFJ7ZV6CYOAIK4DH7KI5A76O5SMQXVQ7ZIEQPHOAIDOAOBCAEFFA, seg_pos={0 0}, upload=529773]: Done uploading piece. err: <nil> Dec 30 19:55:09.609122613 (604h26m9.480205658s) [noise=true , address=145.79.227.33:4887 , piece_id=OVVPGXJQVMJV5UHHGBRS3MM5PB2L4WDVSMW367D6B6KVKJFTEPLQ, seg_pos={0 0}, upload=529773]: Done uploading piece. err: <nil> Dec 30 19:55:09.637107520 (604h26m9.508190555s) [noise=true , address=145.79.227.44:4222 , piece_id=RQ2R7VC6NOSOTK3GHHLISZMPHNJQDXRW5XTVRIQAYWSLGUB2DW6A, seg_pos={0 0}, upload=529773]: Done uploading piece. err: <nil> Dec 30 19:55:09.641172070 (604h26m9.512255115s) [noise=true , address=145.79.227.35:4501 , piece_id=AOTXPNLCU6W27IZXATRTOTE6ZT6UAO53UNFFI6OLGYAGKG4PELPQ, seg_pos={0 0}, upload=529773]: Done uploading piece. err: <nil> Dec 30 19:55:09.653490365 (604h26m9.5245734s) [noise=true , address=145.79.227.49:4362 , piece_id=7OV3IDOSZ4LGEDSGCQQ7K6SSRNCIGKZGWIGGLIHX5TBY4GL2ES5A, seg_pos={0 0}, upload=529773]: Done uploading piece. err: <nil> Dec 30 19:55:09.695230565 (604h26m9.56631361s) [noise=true , address=145.79.227.34:4922 , piece_id=3LFPTHPY5JUSDGVWZPQY46NPEMDXNOGISRR3H2GXERETTNVOS7FQ, seg_pos={0 0}, upload=529773]: Done uploading piece. err: <nil> Dec 30 19:55:09.797550045 (604h26m9.66863308s) [noise=true , address=145.79.227.33:4883 , piece_id=PXIEM4ISCK5XV3UBOEE4ILQM2HFRKRWEEHF7FQLKXUBUD5EFGHTQ, seg_pos={0 0}, upload=529773]: Done uploading piece. err: <nil> Dec 30 19:55:09.797609399 (604h26m9.668692444s) [seg_pos={0 0}, upload=529773]: Segment reached optimal threshold of 14 pieces. Dec 30 19:55:09.798022255 (604h26m9.66910531s) [noise=true , address=145.79.227.40:4055 , piece_id=IQ2D66VK2VWUUFUNMMNP4YEECYE7TITDVKWWCJ6I3LWDBHLD4ATQ, seg_pos={0 0}, upload=529773]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SAWKJT2JEb9ReJUGDZ545Rp7LeQoCJKYsQgmrDA9csufPf8us): protocol: expected piece hash; context canceled; EOF Dec 30 19:55:09.798033045 (604h26m9.66911608s) [noise=true , address=145.79.227.40:4055 , piece_id=IQ2D66VK2VWUUFUNMMNP4YEECYE7TITDVKWWCJ6I3LWDBHLD4ATQ, seg_pos={0 0}, upload=529773]: Long tail cancellation due to optimal threshold Dec 30 19:55:09.798613871 (604h26m9.669696916s) [noise=true , address=145.79.227.18:4024 , piece_id=WVOWBAGISW2SPCKK3EM3DCTDPDAJE6J3YEOAZLMNKNFZMIW6AQAA, seg_pos={0 0}, upload=529773]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bHv9jsVASv3dP3iowvir74PDEHkaaqA3sstiLk7qLSpK2Duzz): protocol: write tcp 188.166.225.170:60354->145.79.227.18:4024: use of closed network connection; write tcp 188.166.225.170:60354->145.79.227.18:4024: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60354->145.79.227.18:4024: use of closed network connection Dec 30 19:55:09.798734340 (604h26m9.669817385s) [noise=true , address=145.79.227.18:4024 , piece_id=WVOWBAGISW2SPCKK3EM3DCTDPDAJE6J3YEOAZLMNKNFZMIW6AQAA, seg_pos={0 0}, upload=529773]: Long tail cancellation due to optimal threshold Dec 30 19:55:09.798916789 (604h26m9.669999834s) [noise=true , address=145.79.227.23:4553 , piece_id=B62LZF2NQJFU6NBFY5BHOYVYOR35PC2UMLNWYRORYGKXR2LRELXA, seg_pos={0 0}, upload=529773]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123VKpEDNfLtYfypf6MD4TsSm7Svho8cdB7iD4HAqefNtC1WgcR): protocol: write tcp 188.166.225.170:55892->145.79.227.23:4553: use of closed network connection; write tcp 188.166.225.170:55892->145.79.227.23:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55892->145.79.227.23:4553: use of closed network connection Dec 30 19:55:09.798925575 (604h26m9.67000862s) [noise=true , address=145.79.227.23:4553 , piece_id=B62LZF2NQJFU6NBFY5BHOYVYOR35PC2UMLNWYRORYGKXR2LRELXA, seg_pos={0 0}, upload=529773]: Long tail cancellation due to optimal threshold Dec 30 19:55:09.799072424 (604h26m9.670155459s) [noise=true , address=45.91.116.4:4944 , piece_id=3AZURY5AJK2PI57ISK4MOGF7EGDMGRA6I3MBKWJPCSB7KK5EJJAQ, seg_pos={0 0}, upload=529773]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jL8XWaamTER1w2SeTtSkikARUpZ3SkrCzfGDQ4FWiXeeRRMws): protocol: expected piece hash; context canceled; EOF Dec 30 19:55:09.799079608 (604h26m9.670162653s) [noise=true , address=45.91.116.4:4944 , piece_id=3AZURY5AJK2PI57ISK4MOGF7EGDMGRA6I3MBKWJPCSB7KK5EJJAQ, seg_pos={0 0}, upload=529773]: Long tail cancellation due to optimal threshold Dec 30 19:55:09.799120526 (604h26m9.670203571s) [seg_pos={0 0}, upload=529773]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 19:55:09.799235074 (604h26m9.67031812s) [upload=529773]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:55:09.799255042 (604h26m9.670338077s) [upload=529773]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:55:10.084827473 (604h26m9.955910528s) [upload=529773]: Done uploading segments... Dec 30 19:55:10.084853432 (604h26m9.955936467s) [upload=529773]: Upload finished. err: <nil> Duration: 1m0.003174446s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..X..XX...X..X.....XX.X.....X.....X.................X...X.X.] duration: 5.997837761s trace: https://jaeger.dev.storj.io/trace/320c3661f90706f2 FAIL (2025-12-30 19:55:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 19:55:20.388852104 (604h26m20.259935149s) [upload=529779]: Starting upload Dec 30 19:55:20.388947735 (604h26m20.26003078s) [upload=529779]: Done starting upload Dec 30 19:55:20.389012809 (604h26m20.260095844s) [upload=529779]: Uploading segments... Dec 30 19:55:20.389225925 (604h26m20.26030899s) [upload=529779]: Got next segment. Inline: false Dec 30 19:55:20.389275039 (604h26m20.260358074s) [upload=529779]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 19:55:20.389319423 (604h26m20.260402458s) [upload=529779]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 19:55:20.637735753 (604h26m20.508818788s) [seg_pos={0 0}, upload=529779]: Begin upload segment... Dec 30 19:55:20.637826036 (604h26m20.508909081s) [seg_pos={0 0}, upload=529779]: Done begin upload segment. Dec 30 19:55:20.637879126 (604h26m20.508962161s) [noise=true , address=145.79.227.41:4086 , piece_id=EFQOIR2WN54XT2QB6QH5BWRBKWMOVGTTZKP7F3TXWNURSHMRZVWQ, seg_pos={0 0}, upload=529779]: Uploading piece... Dec 30 19:55:20.641864566 (604h26m20.512947601s) [noise=true , address=45.91.116.7:4009 , piece_id=KY3KLTLKNAL33VDTYYJ6HT7HAZZYZPAOGARY3P4B2ZNXPQKO433Q, seg_pos={0 0}, upload=529779]: Uploading piece... Dec 30 19:55:20.642527047 (604h26m20.513610092s) [noise=true , address=145.79.227.14:4839 , piece_id=K4CYFRFMWOUOCD5PORP57QU4JGAPY6TNU4G6X7NGYUFS2YTZH4CQ, seg_pos={0 0}, upload=529779]: Uploading piece... Dec 30 19:55:20.642958769 (604h26m20.514041804s) [noise=true , address=145.79.227.46:4601 , piece_id=LVNFGQSBJELIEFP4LLWSXNZZJ3GRZZURLPV2WA7I2RC3IFAP5BDQ, seg_pos={0 0}, upload=529779]: Uploading piece... Dec 30 19:55:20.644658235 (604h26m20.51574127s) [noise=true , address=77.90.31.10:4520 , piece_id=F4YPNFPVVYE4WSVFRQOBY3LRKUG6KQXBRPLKDPR3NSD4ADKRKGVA, seg_pos={0 0}, upload=529779]: Uploading piece... Dec 30 19:55:20.646497668 (604h26m20.517580713s) [noise=true , address=77.90.31.11:4556 , piece_id=BCYHWNQ35V3C3I4OEK2NRARVTSPB7WYZELKW35CZKBYL7GCN2JHQ, seg_pos={0 0}, upload=529779]: Uploading piece... Dec 30 19:55:20.651614881 (604h26m20.522697916s) [noise=true , address=145.79.227.23:4547 , piece_id=6NBHF7VWCWCQ4B6YZ3UGRDGX6UBDCDSSXH44PULJ3JX2XVJTJXWA, seg_pos={0 0}, upload=529779]: Uploading piece... Dec 30 19:55:20.656966772 (604h26m20.528049817s) [noise=true , address=145.79.227.38:4979 , piece_id=FH5E2HAZE76SHEYDPVHX5223STHUZNGURQRGIH35UIVGYWG4HCTQ, seg_pos={0 0}, upload=529779]: Uploading piece... Dec 30 19:55:20.660558651 (604h26m20.531641687s) [noise=true , address=45.91.116.12:4225 , piece_id=Q6LYLKUACD2IKZU3JU3VPRFPMLGPATQXN33XFR57IFKLE2YHQYKQ, seg_pos={0 0}, upload=529779]: Uploading piece... Dec 30 19:55:20.662439884 (604h26m20.533522919s) [noise=true , address=145.79.227.24:4582 , piece_id=IB7JN642RDUUWPERKA76QY6FWAM7GJBLPL2XLQHHLHSEF4IHLBZQ, seg_pos={0 0}, upload=529779]: Uploading piece... Dec 30 19:55:20.666687001 (604h26m20.537770046s) [noise=true , address=45.91.116.13:4156 , piece_id=T4MGN4JZO557LTKCIILPBEZIWOYB6SDGZUOBI2BBTEB2VI7TBC4A, seg_pos={0 0}, upload=529779]: Uploading piece... Dec 30 19:55:20.668820473 (604h26m20.539903518s) [noise=true , address=45.91.116.14:4190 , piece_id=BIWL5WYF3KQGS33YW3BBPSWZSPEIAAJHVVM4HBKKWNB4QI63KCAQ, seg_pos={0 0}, upload=529779]: Uploading piece... Dec 30 19:55:20.669418572 (604h26m20.540501617s) [noise=true , address=45.91.116.10:4081 , piece_id=GB5RS4FDH3NS2VJBHSZOZGD62VEVPF6JHLQLGDUP2JAG2MDIBOEQ, seg_pos={0 0}, upload=529779]: Uploading piece... Dec 30 19:55:20.673339789 (604h26m20.544422824s) [noise=true , address=77.90.31.21:4878 , piece_id=H5KQHD4EXZBWAAEMRTWUVOKSDG7DJNIVW2SS2YNPJULPI75WSI4Q, seg_pos={0 0}, upload=529779]: Uploading piece... Dec 30 19:55:20.674871596 (604h26m20.545954631s) [noise=true , address=145.79.227.18:4007 , piece_id=D4DGMLJZJSIW274RIPKRL7CQJEJBQWKJYDMM76TJHKIZEWD6DRNQ, seg_pos={0 0}, upload=529779]: Uploading piece... Dec 30 19:55:20.676018379 (604h26m20.547101414s) [noise=true , address=45.91.116.4:4940 , piece_id=2UJYMIHGJK7K3SOIQULVYKBYOXRQA5422D5BG2U3IWD5MLGRHBKA, seg_pos={0 0}, upload=529779]: Uploading piece... Dec 30 19:55:22.275159914 (604h26m22.146242959s) [upload=529779]: Next returned nil segment Dec 30 19:55:22.275172628 (604h26m22.146255663s) [upload=529779]: Waiting for error group managing segments... Dec 30 19:55:22.648941708 (604h26m22.520024743s) [noise=true , address=145.79.227.46:4601 , piece_id=LVNFGQSBJELIEFP4LLWSXNZZJ3GRZZURLPV2WA7I2RC3IFAP5BDQ, seg_pos={0 0}, upload=529779]: Done uploading piece. err: <nil> Dec 30 19:55:22.655588274 (604h26m22.526671319s) [noise=true , address=145.79.227.38:4979 , piece_id=FH5E2HAZE76SHEYDPVHX5223STHUZNGURQRGIH35UIVGYWG4HCTQ, seg_pos={0 0}, upload=529779]: Done uploading piece. err: <nil> Dec 30 19:55:22.666706344 (604h26m22.537789379s) [noise=true , address=145.79.227.23:4547 , piece_id=6NBHF7VWCWCQ4B6YZ3UGRDGX6UBDCDSSXH44PULJ3JX2XVJTJXWA, seg_pos={0 0}, upload=529779]: Done uploading piece. err: <nil> Dec 30 19:55:22.670664591 (604h26m22.541747637s) [noise=true , address=145.79.227.18:4007 , piece_id=D4DGMLJZJSIW274RIPKRL7CQJEJBQWKJYDMM76TJHKIZEWD6DRNQ, seg_pos={0 0}, upload=529779]: Done uploading piece. err: <nil> Dec 30 19:55:22.671277638 (604h26m22.542360673s) [noise=true , address=145.79.227.24:4582 , piece_id=IB7JN642RDUUWPERKA76QY6FWAM7GJBLPL2XLQHHLHSEF4IHLBZQ, seg_pos={0 0}, upload=529779]: Done uploading piece. err: <nil> Dec 30 19:55:22.755004005 (604h26m22.626087041s) [noise=true , address=145.79.227.41:4086 , piece_id=EFQOIR2WN54XT2QB6QH5BWRBKWMOVGTTZKP7F3TXWNURSHMRZVWQ, seg_pos={0 0}, upload=529779]: Done uploading piece. err: <nil> Dec 30 19:55:22.763009529 (604h26m22.634092574s) [noise=true , address=145.79.227.14:4839 , piece_id=K4CYFRFMWOUOCD5PORP57QU4JGAPY6TNU4G6X7NGYUFS2YTZH4CQ, seg_pos={0 0}, upload=529779]: Done uploading piece. err: <nil> Dec 30 19:55:22.844670152 (604h26m22.715753198s) [noise=true , address=77.90.31.21:4878 , piece_id=H5KQHD4EXZBWAAEMRTWUVOKSDG7DJNIVW2SS2YNPJULPI75WSI4Q, seg_pos={0 0}, upload=529779]: Done uploading piece. err: <nil> Dec 30 19:55:22.869369905 (604h26m22.74045294s) [noise=true , address=77.90.31.10:4520 , piece_id=F4YPNFPVVYE4WSVFRQOBY3LRKUG6KQXBRPLKDPR3NSD4ADKRKGVA, seg_pos={0 0}, upload=529779]: Done uploading piece. err: <nil> Dec 30 19:55:22.880892005 (604h26m22.75197505s) [noise=true , address=77.90.31.11:4556 , piece_id=BCYHWNQ35V3C3I4OEK2NRARVTSPB7WYZELKW35CZKBYL7GCN2JHQ, seg_pos={0 0}, upload=529779]: Done uploading piece. err: <nil> Dec 30 19:55:22.959345230 (604h26m22.830428265s) [noise=true , address=45.91.116.12:4225 , piece_id=Q6LYLKUACD2IKZU3JU3VPRFPMLGPATQXN33XFR57IFKLE2YHQYKQ, seg_pos={0 0}, upload=529779]: Done uploading piece. err: <nil> Dec 30 19:55:22.964490528 (604h26m22.835573573s) [noise=true , address=45.91.116.4:4940 , piece_id=2UJYMIHGJK7K3SOIQULVYKBYOXRQA5422D5BG2U3IWD5MLGRHBKA, seg_pos={0 0}, upload=529779]: Done uploading piece. err: <nil> Dec 30 19:55:22.986740726 (604h26m22.857823771s) [noise=true , address=45.91.116.10:4081 , piece_id=GB5RS4FDH3NS2VJBHSZOZGD62VEVPF6JHLQLGDUP2JAG2MDIBOEQ, seg_pos={0 0}, upload=529779]: Done uploading piece. err: <nil> Dec 30 19:55:22.986767296 (604h26m22.857850351s) [seg_pos={0 0}, upload=529779]: Segment reached optimal threshold of 13 pieces. Dec 30 19:55:22.987002103 (604h26m22.858085138s) [noise=true , address=45.91.116.13:4156 , piece_id=T4MGN4JZO557LTKCIILPBEZIWOYB6SDGZUOBI2BBTEB2VI7TBC4A, seg_pos={0 0}, upload=529779]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 30 19:55:22.987009607 (604h26m22.858092642s) [noise=true , address=45.91.116.13:4156 , piece_id=T4MGN4JZO557LTKCIILPBEZIWOYB6SDGZUOBI2BBTEB2VI7TBC4A, seg_pos={0 0}, upload=529779]: Long tail cancellation due to optimal threshold Dec 30 19:55:22.987141499 (604h26m22.858224544s) [noise=true , address=45.91.116.14:4190 , piece_id=BIWL5WYF3KQGS33YW3BBPSWZSPEIAAJHVVM4HBKKWNB4QI63KCAQ, seg_pos={0 0}, upload=529779]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 30 19:55:22.987149864 (604h26m22.8582329s) [noise=true , address=45.91.116.14:4190 , piece_id=BIWL5WYF3KQGS33YW3BBPSWZSPEIAAJHVVM4HBKKWNB4QI63KCAQ, seg_pos={0 0}, upload=529779]: Long tail cancellation due to optimal threshold Dec 30 19:55:22.987217473 (604h26m22.858300508s) [noise=true , address=45.91.116.7:4009 , piece_id=KY3KLTLKNAL33VDTYYJ6HT7HAZZYZPAOGARY3P4B2ZNXPQKO433Q, seg_pos={0 0}, upload=529779]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 30 19:55:22.987224056 (604h26m22.858307091s) [noise=true , address=45.91.116.7:4009 , piece_id=KY3KLTLKNAL33VDTYYJ6HT7HAZZYZPAOGARY3P4B2ZNXPQKO433Q, seg_pos={0 0}, upload=529779]: Long tail cancellation due to optimal threshold Dec 30 19:55:22.987240357 (604h26m22.858323402s) [seg_pos={0 0}, upload=529779]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 19:55:22.987312464 (604h26m22.858395499s) [upload=529779]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:55:22.987319037 (604h26m22.858402082s) [upload=529779]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:55:23.280452495 (604h26m23.15153553s) [upload=529779]: Done uploading segments... Dec 30 19:55:23.280474928 (604h26m23.151557973s) [upload=529779]: Upload finished. err: <nil> Duration: 5.997856527s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...X..XX...X..X.....XX.X.....X.....X.................X...X.X] duration: 30.002839635s trace: https://jaeger.dev.storj.io/trace/320c3661f90706f2 FAIL (2025-12-30 19:55:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 19:55:20.388852104 (604h26m20.259935149s) [upload=529779]: Starting upload Dec 30 19:55:20.388947735 (604h26m20.26003078s) [upload=529779]: Done starting upload Dec 30 19:55:20.389012809 (604h26m20.260095844s) [upload=529779]: Uploading segments... Dec 30 19:55:20.389225925 (604h26m20.26030899s) [upload=529779]: Got next segment. Inline: false Dec 30 19:55:20.389275039 (604h26m20.260358074s) [upload=529779]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 19:55:20.389319423 (604h26m20.260402458s) [upload=529779]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 19:55:20.637735753 (604h26m20.508818788s) [seg_pos={0 0}, upload=529779]: Begin upload segment... Dec 30 19:55:20.637826036 (604h26m20.508909081s) [seg_pos={0 0}, upload=529779]: Done begin upload segment. Dec 30 19:55:20.637879126 (604h26m20.508962161s) [noise=true , address=145.79.227.41:4086 , piece_id=EFQOIR2WN54XT2QB6QH5BWRBKWMOVGTTZKP7F3TXWNURSHMRZVWQ, seg_pos={0 0}, upload=529779]: Uploading piece... Dec 30 19:55:20.641864566 (604h26m20.512947601s) [noise=true , address=45.91.116.7:4009 , piece_id=KY3KLTLKNAL33VDTYYJ6HT7HAZZYZPAOGARY3P4B2ZNXPQKO433Q, seg_pos={0 0}, upload=529779]: Uploading piece... Dec 30 19:55:20.642527047 (604h26m20.513610092s) [noise=true , address=145.79.227.14:4839 , piece_id=K4CYFRFMWOUOCD5PORP57QU4JGAPY6TNU4G6X7NGYUFS2YTZH4CQ, seg_pos={0 0}, upload=529779]: Uploading piece... Dec 30 19:55:20.642958769 (604h26m20.514041804s) [noise=true , address=145.79.227.46:4601 , piece_id=LVNFGQSBJELIEFP4LLWSXNZZJ3GRZZURLPV2WA7I2RC3IFAP5BDQ, seg_pos={0 0}, upload=529779]: Uploading piece... Dec 30 19:55:20.644658235 (604h26m20.51574127s) [noise=true , address=77.90.31.10:4520 , piece_id=F4YPNFPVVYE4WSVFRQOBY3LRKUG6KQXBRPLKDPR3NSD4ADKRKGVA, seg_pos={0 0}, upload=529779]: Uploading piece... Dec 30 19:55:20.646497668 (604h26m20.517580713s) [noise=true , address=77.90.31.11:4556 , piece_id=BCYHWNQ35V3C3I4OEK2NRARVTSPB7WYZELKW35CZKBYL7GCN2JHQ, seg_pos={0 0}, upload=529779]: Uploading piece... Dec 30 19:55:20.651614881 (604h26m20.522697916s) [noise=true , address=145.79.227.23:4547 , piece_id=6NBHF7VWCWCQ4B6YZ3UGRDGX6UBDCDSSXH44PULJ3JX2XVJTJXWA, seg_pos={0 0}, upload=529779]: Uploading piece... Dec 30 19:55:20.656966772 (604h26m20.528049817s) [noise=true , address=145.79.227.38:4979 , piece_id=FH5E2HAZE76SHEYDPVHX5223STHUZNGURQRGIH35UIVGYWG4HCTQ, seg_pos={0 0}, upload=529779]: Uploading piece... Dec 30 19:55:20.660558651 (604h26m20.531641687s) [noise=true , address=45.91.116.12:4225 , piece_id=Q6LYLKUACD2IKZU3JU3VPRFPMLGPATQXN33XFR57IFKLE2YHQYKQ, seg_pos={0 0}, upload=529779]: Uploading piece... Dec 30 19:55:20.662439884 (604h26m20.533522919s) [noise=true , address=145.79.227.24:4582 , piece_id=IB7JN642RDUUWPERKA76QY6FWAM7GJBLPL2XLQHHLHSEF4IHLBZQ, seg_pos={0 0}, upload=529779]: Uploading piece... Dec 30 19:55:20.666687001 (604h26m20.537770046s) [noise=true , address=45.91.116.13:4156 , piece_id=T4MGN4JZO557LTKCIILPBEZIWOYB6SDGZUOBI2BBTEB2VI7TBC4A, seg_pos={0 0}, upload=529779]: Uploading piece... Dec 30 19:55:20.668820473 (604h26m20.539903518s) [noise=true , address=45.91.116.14:4190 , piece_id=BIWL5WYF3KQGS33YW3BBPSWZSPEIAAJHVVM4HBKKWNB4QI63KCAQ, seg_pos={0 0}, upload=529779]: Uploading piece... Dec 30 19:55:20.669418572 (604h26m20.540501617s) [noise=true , address=45.91.116.10:4081 , piece_id=GB5RS4FDH3NS2VJBHSZOZGD62VEVPF6JHLQLGDUP2JAG2MDIBOEQ, seg_pos={0 0}, upload=529779]: Uploading piece... Dec 30 19:55:20.673339789 (604h26m20.544422824s) [noise=true , address=77.90.31.21:4878 , piece_id=H5KQHD4EXZBWAAEMRTWUVOKSDG7DJNIVW2SS2YNPJULPI75WSI4Q, seg_pos={0 0}, upload=529779]: Uploading piece... Dec 30 19:55:20.674871596 (604h26m20.545954631s) [noise=true , address=145.79.227.18:4007 , piece_id=D4DGMLJZJSIW274RIPKRL7CQJEJBQWKJYDMM76TJHKIZEWD6DRNQ, seg_pos={0 0}, upload=529779]: Uploading piece... Dec 30 19:55:20.676018379 (604h26m20.547101414s) [noise=true , address=45.91.116.4:4940 , piece_id=2UJYMIHGJK7K3SOIQULVYKBYOXRQA5422D5BG2U3IWD5MLGRHBKA, seg_pos={0 0}, upload=529779]: Uploading piece... Dec 30 19:55:22.275159914 (604h26m22.146242959s) [upload=529779]: Next returned nil segment Dec 30 19:55:22.275172628 (604h26m22.146255663s) [upload=529779]: Waiting for error group managing segments... Dec 30 19:55:22.648941708 (604h26m22.520024743s) [noise=true , address=145.79.227.46:4601 , piece_id=LVNFGQSBJELIEFP4LLWSXNZZJ3GRZZURLPV2WA7I2RC3IFAP5BDQ, seg_pos={0 0}, upload=529779]: Done uploading piece. err: <nil> Dec 30 19:55:22.655588274 (604h26m22.526671319s) [noise=true , address=145.79.227.38:4979 , piece_id=FH5E2HAZE76SHEYDPVHX5223STHUZNGURQRGIH35UIVGYWG4HCTQ, seg_pos={0 0}, upload=529779]: Done uploading piece. err: <nil> Dec 30 19:55:22.666706344 (604h26m22.537789379s) [noise=true , address=145.79.227.23:4547 , piece_id=6NBHF7VWCWCQ4B6YZ3UGRDGX6UBDCDSSXH44PULJ3JX2XVJTJXWA, seg_pos={0 0}, upload=529779]: Done uploading piece. err: <nil> Dec 30 19:55:22.670664591 (604h26m22.541747637s) [noise=true , address=145.79.227.18:4007 , piece_id=D4DGMLJZJSIW274RIPKRL7CQJEJBQWKJYDMM76TJHKIZEWD6DRNQ, seg_pos={0 0}, upload=529779]: Done uploading piece. err: <nil> Dec 30 19:55:22.671277638 (604h26m22.542360673s) [noise=true , address=145.79.227.24:4582 , piece_id=IB7JN642RDUUWPERKA76QY6FWAM7GJBLPL2XLQHHLHSEF4IHLBZQ, seg_pos={0 0}, upload=529779]: Done uploading piece. err: <nil> Dec 30 19:55:22.755004005 (604h26m22.626087041s) [noise=true , address=145.79.227.41:4086 , piece_id=EFQOIR2WN54XT2QB6QH5BWRBKWMOVGTTZKP7F3TXWNURSHMRZVWQ, seg_pos={0 0}, upload=529779]: Done uploading piece. err: <nil> Dec 30 19:55:22.763009529 (604h26m22.634092574s) [noise=true , address=145.79.227.14:4839 , piece_id=K4CYFRFMWOUOCD5PORP57QU4JGAPY6TNU4G6X7NGYUFS2YTZH4CQ, seg_pos={0 0}, upload=529779]: Done uploading piece. err: <nil> Dec 30 19:55:22.844670152 (604h26m22.715753198s) [noise=true , address=77.90.31.21:4878 , piece_id=H5KQHD4EXZBWAAEMRTWUVOKSDG7DJNIVW2SS2YNPJULPI75WSI4Q, seg_pos={0 0}, upload=529779]: Done uploading piece. err: <nil> Dec 30 19:55:22.869369905 (604h26m22.74045294s) [noise=true , address=77.90.31.10:4520 , piece_id=F4YPNFPVVYE4WSVFRQOBY3LRKUG6KQXBRPLKDPR3NSD4ADKRKGVA, seg_pos={0 0}, upload=529779]: Done uploading piece. err: <nil> Dec 30 19:55:22.880892005 (604h26m22.75197505s) [noise=true , address=77.90.31.11:4556 , piece_id=BCYHWNQ35V3C3I4OEK2NRARVTSPB7WYZELKW35CZKBYL7GCN2JHQ, seg_pos={0 0}, upload=529779]: Done uploading piece. err: <nil> Dec 30 19:55:22.959345230 (604h26m22.830428265s) [noise=true , address=45.91.116.12:4225 , piece_id=Q6LYLKUACD2IKZU3JU3VPRFPMLGPATQXN33XFR57IFKLE2YHQYKQ, seg_pos={0 0}, upload=529779]: Done uploading piece. err: <nil> Dec 30 19:55:22.964490528 (604h26m22.835573573s) [noise=true , address=45.91.116.4:4940 , piece_id=2UJYMIHGJK7K3SOIQULVYKBYOXRQA5422D5BG2U3IWD5MLGRHBKA, seg_pos={0 0}, upload=529779]: Done uploading piece. err: <nil> Dec 30 19:55:22.986740726 (604h26m22.857823771s) [noise=true , address=45.91.116.10:4081 , piece_id=GB5RS4FDH3NS2VJBHSZOZGD62VEVPF6JHLQLGDUP2JAG2MDIBOEQ, seg_pos={0 0}, upload=529779]: Done uploading piece. err: <nil> Dec 30 19:55:22.986767296 (604h26m22.857850351s) [seg_pos={0 0}, upload=529779]: Segment reached optimal threshold of 13 pieces. Dec 30 19:55:22.987002103 (604h26m22.858085138s) [noise=true , address=45.91.116.13:4156 , piece_id=T4MGN4JZO557LTKCIILPBEZIWOYB6SDGZUOBI2BBTEB2VI7TBC4A, seg_pos={0 0}, upload=529779]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 30 19:55:22.987009607 (604h26m22.858092642s) [noise=true , address=45.91.116.13:4156 , piece_id=T4MGN4JZO557LTKCIILPBEZIWOYB6SDGZUOBI2BBTEB2VI7TBC4A, seg_pos={0 0}, upload=529779]: Long tail cancellation due to optimal threshold Dec 30 19:55:22.987141499 (604h26m22.858224544s) [noise=true , address=45.91.116.14:4190 , piece_id=BIWL5WYF3KQGS33YW3BBPSWZSPEIAAJHVVM4HBKKWNB4QI63KCAQ, seg_pos={0 0}, upload=529779]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 30 19:55:22.987149864 (604h26m22.8582329s) [noise=true , address=45.91.116.14:4190 , piece_id=BIWL5WYF3KQGS33YW3BBPSWZSPEIAAJHVVM4HBKKWNB4QI63KCAQ, seg_pos={0 0}, upload=529779]: Long tail cancellation due to optimal threshold Dec 30 19:55:22.987217473 (604h26m22.858300508s) [noise=true , address=45.91.116.7:4009 , piece_id=KY3KLTLKNAL33VDTYYJ6HT7HAZZYZPAOGARY3P4B2ZNXPQKO433Q, seg_pos={0 0}, upload=529779]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 30 19:55:22.987224056 (604h26m22.858307091s) [noise=true , address=45.91.116.7:4009 , piece_id=KY3KLTLKNAL33VDTYYJ6HT7HAZZYZPAOGARY3P4B2ZNXPQKO433Q, seg_pos={0 0}, upload=529779]: Long tail cancellation due to optimal threshold Dec 30 19:55:22.987240357 (604h26m22.858323402s) [seg_pos={0 0}, upload=529779]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 19:55:22.987312464 (604h26m22.858395499s) [upload=529779]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:55:22.987319037 (604h26m22.858402082s) [upload=529779]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:55:23.280452495 (604h26m23.15153553s) [upload=529779]: Done uploading segments... Dec 30 19:55:23.280474928 (604h26m23.151557973s) [upload=529779]: Upload finished. err: <nil> Duration: 30.002861276s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [....X..XX...X..X.....XX.X.....X.....X.................X...X.] duration: 27.175518757s trace: https://jaeger.dev.storj.io/trace/62ea6ecd955e2692 FAIL (2025-12-30 19:54:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: WTJIIOZZJW5IXIFORC2BANV5S6GSZX3A6VCCMWCFTBDTCFAWB5VA) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: 72IJIO3E2WUUYDFDNBREZFSQVI5F772GM5X4XZZPQUWHOJLPK2QQ) context deadline exceeded ======LOGS====== 29761]: Starting upload Dec 30 19:54:00.971827755 (604h25m0.84291082s) [upload=529761]: Done starting upload Dec 30 19:54:00.972046472 (604h25m0.843129507s) [upload=529761]: Uploading segments... Dec 30 19:54:00.972095655 (604h25m0.843178711s) [upload=529761]: Got next segment. Inline: false Dec 30 19:54:00.972126975 (604h25m0.84321001s) [upload=529761]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 19:54:00.972132165 (604h25m0.84321524s) [upload=529761]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 19:54:01.217503947 (604h25m1.088586992s) [seg_pos={0 0}, upload=529761]: Begin upload segment... Dec 30 19:54:01.217570885 (604h25m1.08865393s) [seg_pos={0 0}, upload=529761]: Done begin upload segment. Dec 30 19:54:01.217605882 (604h25m1.088688937s) [noise=true , address=145.79.227.20:4437 , piece_id=ALTKEV25PB4XQGW7PYPGP7LXOBOJTBHE75VF6UKZLZA7NBIZRGKA, seg_pos={0 0}, upload=529761]: Uploading piece... Dec 30 19:54:01.217636811 (604h25m1.088719856s) [noise=true , address=45.91.116.5:4300 , piece_id=TC24DWZBNLBC3ZHABNXSN23NMVYCHJ3VHM3W7FSODQVKL4RLPJ6A, seg_pos={0 0}, upload=529761]: Uploading piece... Dec 30 19:54:01.217692477 (604h25m1.088775522s) [noise=true , address=145.79.227.17:4971 , piece_id=BBYHYMT6H2SI6XJIO3KO7PBX6EK46DUA4CAMM2VWPC57STOZLGFA, seg_pos={0 0}, upload=529761]: Uploading piece... Dec 30 19:54:01.218460690 (604h25m1.089543735s) [noise=true , address=77.90.31.17:4773 , piece_id=IV3VOWAQBVKIQ7MWYPE7PDZTEEARX555G6RMC7XJSNCU7XCKKCFQ, seg_pos={0 0}, upload=529761]: Uploading piece... Dec 30 19:54:01.218728860 (604h25m1.089811905s) [noise=true , address=45.91.116.9:4263 , piece_id=S77ZC7VREF5IF24QUXDI4SFYTOG4AVUHS76NMCL62KTM35OIBHBA, seg_pos={0 0}, upload=529761]: Uploading piece... Dec 30 19:54:01.218999104 (604h25m1.09008215s) [noise=true , address=145.79.227.39:4014 , piece_id=HFT4NOZ5RIEUTC3DLCQR3Z6JUGUGUVH4BVR3XYDDWUAL432MYP4A, seg_pos={0 0}, upload=529761]: Uploading piece... Dec 30 19:54:01.219110657 (604h25m1.090193692s) [noise=true , address=45.91.116.12:4226 , piece_id=Y4IAQFVBUR6JXLGWLXHT7DKQTJGXY36DKEKCPRKBMIWWKZ4NL6DA, seg_pos={0 0}, upload=529761]: Uploading piece... Dec 30 19:54:01.219549753 (604h25m1.090632798s) [noise=true , address=77.90.31.10:4520 , piece_id=WTJIIOZZJW5IXIFORC2BANV5S6GSZX3A6VCCMWCFTBDTCFAWB5VA, seg_pos={0 0}, upload=529761]: Uploading piece... Dec 30 19:54:01.219933063 (604h25m1.091016108s) [noise=true , address=145.79.227.15:4876 , piece_id=XABRDRBTOTGCBUUZDPJ7I35ZTSE33ATMFUYWKXB3EAAIAK235BKA, seg_pos={0 0}, upload=529761]: Uploading piece... Dec 30 19:54:01.220144646 (604h25m1.091227691s) [noise=true , address=145.79.227.36:4517 , piece_id=FNKAM5YZKOGHRUUP63V4IQ53OSIIORPCWJ4UOWSTD3DBAPYV3BHA, seg_pos={0 0}, upload=529761]: Uploading piece... Dec 30 19:54:01.220322094 (604h25m1.091405139s) [noise=true , address=77.90.31.15:4700 , piece_id=VMCFO2PPX77NLE6E3KX434M2BK26RBBGMUG775HF2MJZS2GOH2WQ, seg_pos={0 0}, upload=529761]: Uploading piece... Dec 30 19:54:01.220824331 (604h25m1.091907366s) [noise=true , address=45.91.116.11:4118 , piece_id=YD4RQAOP33CDDPNJKGW6AB3X47VXKYJBIPBI4DRVMQ2XGTEJSHYA, seg_pos={0 0}, upload=529761]: Uploading piece... Dec 30 19:54:01.221399044 (604h25m1.09248209s) [noise=true , address=145.79.227.23:4547 , piece_id=6G7YQF2SWESD3RWSEUZAG5RCCWJKQVZD4X4XUXYJHDZYZ4H6RVYQ, seg_pos={0 0}, upload=529761]: Uploading piece... Dec 30 19:54:01.221996632 (604h25m1.093079677s) [noise=true , address=145.79.227.51:4411 , piece_id=DDNNYRQ24RVYFXM7FIYCFWHOCEMAX66NVPJ4SWDQQ7WKBLU54QFA, seg_pos={0 0}, upload=529761]: Uploading piece... Dec 30 19:54:01.222447932 (604h25m1.093530977s) [noise=true , address=45.91.116.13:4155 , piece_id=72IJIO3E2WUUYDFDNBREZFSQVI5F772GM5X4XZZPQUWHOJLPK2QQ, seg_pos={0 0}, upload=529761]: Uploading piece... Dec 30 19:54:01.222795834 (604h25m1.093878879s) [noise=true , address=77.90.31.11:4555 , piece_id=4KQCJHSS3ED4NL2QIO7XG7VX5P3NBKO3JWFT565XBCNKKCVRHQOA, seg_pos={0 0}, upload=529761]: Uploading piece... Dec 30 19:54:02.814372916 (604h25m2.685455961s) [upload=529761]: Next returned nil segment Dec 30 19:54:02.814386151 (604h25m2.685469186s) [upload=529761]: Waiting for error group managing segments... Dec 30 19:54:03.191559262 (604h25m3.062642307s) [noise=true , address=145.79.227.17:4971 , piece_id=BBYHYMT6H2SI6XJIO3KO7PBX6EK46DUA4CAMM2VWPC57STOZLGFA, seg_pos={0 0}, upload=529761]: Done uploading piece. err: <nil> Dec 30 19:54:03.196306852 (604h25m3.067389897s) [noise=true , address=145.79.227.23:4547 , piece_id=6G7YQF2SWESD3RWSEUZAG5RCCWJKQVZD4X4XUXYJHDZYZ4H6RVYQ, seg_pos={0 0}, upload=529761]: Done uploading piece. err: <nil> Dec 30 19:54:03.197469086 (604h25m3.068552121s) [noise=true , address=145.79.227.51:4411 , piece_id=DDNNYRQ24RVYFXM7FIYCFWHOCEMAX66NVPJ4SWDQQ7WKBLU54QFA, seg_pos={0 0}, upload=529761]: Done uploading piece. err: <nil> Dec 30 19:54:03.212497021 (604h25m3.083580056s) [noise=true , address=145.79.227.20:4437 , piece_id=ALTKEV25PB4XQGW7PYPGP7LXOBOJTBHE75VF6UKZLZA7NBIZRGKA, seg_pos={0 0}, upload=529761]: Done uploading piece. err: <nil> Dec 30 19:54:03.212905800 (604h25m3.083988845s) [noise=true , address=145.79.227.15:4876 , piece_id=XABRDRBTOTGCBUUZDPJ7I35ZTSE33ATMFUYWKXB3EAAIAK235BKA, seg_pos={0 0}, upload=529761]: Done uploading piece. err: <nil> Dec 30 19:54:03.213920081 (604h25m3.085003116s) [noise=true , address=145.79.227.39:4014 , piece_id=HFT4NOZ5RIEUTC3DLCQR3Z6JUGUGUVH4BVR3XYDDWUAL432MYP4A, seg_pos={0 0}, upload=529761]: Done uploading piece. err: <nil> Dec 30 19:54:03.227126788 (604h25m3.098209833s) [noise=true , address=145.79.227.36:4517 , piece_id=FNKAM5YZKOGHRUUP63V4IQ53OSIIORPCWJ4UOWSTD3DBAPYV3BHA, seg_pos={0 0}, upload=529761]: Done uploading piece. err: <nil> Dec 30 19:54:03.396500439 (604h25m3.267583485s) [noise=true , address=77.90.31.10:4520 , piece_id=WTJIIOZZJW5IXIFORC2BANV5S6GSZX3A6VCCMWCFTBDTCFAWB5VA, seg_pos={0 0}, upload=529761]: Done uploading piece. err: <nil> Dec 30 19:54:03.516423741 (604h25m3.387506776s) [noise=true , address=77.90.31.15:4700 , piece_id=VMCFO2PPX77NLE6E3KX434M2BK26RBBGMUG775HF2MJZS2GOH2WQ, seg_pos={0 0}, upload=529761]: Done uploading piece. err: <nil> Dec 30 19:54:03.548322150 (604h25m3.419405195s) [noise=true , address=45.91.116.9:4263 , piece_id=S77ZC7VREF5IF24QUXDI4SFYTOG4AVUHS76NMCL62KTM35OIBHBA, seg_pos={0 0}, upload=529761]: Done uploading piece. err: <nil> Dec 30 19:54:03.578983784 (604h25m3.450066829s) [noise=true , address=45.91.116.12:4226 , piece_id=Y4IAQFVBUR6JXLGWLXHT7DKQTJGXY36DKEKCPRKBMIWWKZ4NL6DA, seg_pos={0 0}, upload=529761]: Done uploading piece. err: <nil> Dec 30 19:54:03.658414372 (604h25m3.529497417s) [noise=true , address=45.91.116.13:4155 , piece_id=72IJIO3E2WUUYDFDNBREZFSQVI5F772GM5X4XZZPQUWHOJLPK2QQ, seg_pos={0 0}, upload=529761]: Done uploading piece. err: <nil> Dec 30 19:54:03.672846182 (604h25m3.543929227s) [noise=true , address=45.91.116.5:4300 , piece_id=TC24DWZBNLBC3ZHABNXSN23NMVYCHJ3VHM3W7FSODQVKL4RLPJ6A, seg_pos={0 0}, upload=529761]: Done uploading piece. err: <nil> Dec 30 19:54:03.672871320 (604h25m3.543954365s) [seg_pos={0 0}, upload=529761]: Segment reached optimal threshold of 13 pieces. Dec 30 19:54:03.673113350 (604h25m3.544196395s) [noise=true , address=45.91.116.11:4118 , piece_id=YD4RQAOP33CDDPNJKGW6AB3X47VXKYJBIPBI4DRVMQ2XGTEJSHYA, seg_pos={0 0}, upload=529761]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:54692->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:54692->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54692->45.91.116.11:4118: use of closed network connection Dec 30 19:54:03.673121727 (604h25m3.544204772s) [noise=true , address=45.91.116.11:4118 , piece_id=YD4RQAOP33CDDPNJKGW6AB3X47VXKYJBIPBI4DRVMQ2XGTEJSHYA, seg_pos={0 0}, upload=529761]: Long tail cancellation due to optimal threshold Dec 30 19:54:03.673344661 (604h25m3.544427706s) [noise=true , address=77.90.31.11:4555 , piece_id=4KQCJHSS3ED4NL2QIO7XG7VX5P3NBKO3JWFT565XBCNKKCVRHQOA, seg_pos={0 0}, upload=529761]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 30 19:54:03.673361774 (604h25m3.544444819s) [noise=true , address=77.90.31.11:4555 , piece_id=4KQCJHSS3ED4NL2QIO7XG7VX5P3NBKO3JWFT565XBCNKKCVRHQOA, seg_pos={0 0}, upload=529761]: Long tail cancellation due to optimal threshold Dec 30 19:54:03.673459760 (604h25m3.544542795s) [noise=true , address=77.90.31.17:4773 , piece_id=IV3VOWAQBVKIQ7MWYPE7PDZTEEARX555G6RMC7XJSNCU7XCKKCFQ, seg_pos={0 0}, upload=529761]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: read tcp 188.166.225.170:41572->77.90.31.17:4773: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 19:54:03.673469980 (604h25m3.544553025s) [noise=true , address=77.90.31.17:4773 , piece_id=IV3VOWAQBVKIQ7MWYPE7PDZTEEARX555G6RMC7XJSNCU7XCKKCFQ, seg_pos={0 0}, upload=529761]: Long tail cancellation due to optimal threshold Dec 30 19:54:03.673494726 (604h25m3.544577771s) [seg_pos={0 0}, upload=529761]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 19:54:03.673583836 (604h25m3.544666882s) [upload=529761]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:54:03.673590399 (604h25m3.544673444s) [upload=529761]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:54:03.978363310 (604h25m3.849446355s) [upload=529761]: Done uploading segments... Dec 30 19:54:03.978384961 (604h25m3.849468007s) [upload=529761]: Upload finished. err: <nil> Duration: 27.175535018s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X..X.....................X......................X....] duration: 15.402166932s trace: https://jaeger.dev.storj.io/trace/fd3a56936ae71f2 FAIL (2025-12-30 19:54:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== piece_id=AMJKLUFDIY4M65Q4Y3DENPIQ2LP7IQDIBB5EJSXCMEE2MKPUOEDA, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.407338347 (604h22m15.278421392s) [noise=true , address=77.90.31.10:4526 , piece_id=EGGUMVBH572S2LZZWUYBKX2JSH2NVXC2BLHLXOKFUZKDMXHNPG4A, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.407626666 (604h22m15.278709711s) [noise=true , address=145.79.227.35:4487 , piece_id=YHVVDBISXCSLG3UW5XT7QEAKXOHSPS7RUY6HKZZS7NITA7OV47TQ, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.407809223 (604h22m15.278892258s) [noise=true , address=145.79.227.39:4040 , piece_id=OYYRO2KDG43DP2XYKFHWLSUJMSKQ6VC57GSW2PRNIA42XUTHEC2A, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.408000178 (604h22m15.279083213s) [noise=true , address=145.79.227.22:4532 , piece_id=34CFPGEIUKJAWIOHGCYT2IPARDF2Q6V4AZ75OEYJKWSGJP4XBLAA, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.408215778 (604h22m15.279298813s) [noise=true , address=77.90.31.16:4755 , piece_id=HSEFLVXWFD73XZ6FVA6L33OIZ4XMUD6KIC7OMUV2TZKSDJHBZIMQ, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.408338902 (604h22m15.279421937s) [noise=true , address=145.79.227.14:4857 , piece_id=UAJPWWOJNMKWIR664WTJJTNLW6BSBWAXMBTM2KP27ZNDX5XUV7HA, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.409170024 (604h22m15.28025306s) [noise=true , address=193.149.164.143:4704 , piece_id=LIPJT75FNGVN73FY277IE5FH6PXESYJXBCIUEVXJFQJ36QEHTH7A, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.410288113 (604h22m15.281371148s) [noise=true , address=145.79.227.34:4937 , piece_id=GRDKPX53C5IPFCL2SE6L3NFCJRXUC7FPJ32HTFCCQGIIRUIITJ6Q, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.411612275 (604h22m15.28269531s) [noise=true , address=45.91.116.8:4049 , piece_id=CDS23XTX2ILZKX7CWONNQ34F7W6G4BQHZOZLKFCWYJQKM3COPF4A, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.413868601 (604h22m15.284951646s) [noise=true , address=45.91.116.8:4073 , piece_id=EV3LO3HZRKQF4A5R3HSPNCXKRE6BYLIQ2BQW6R5LMR4ROQQ6X3OA, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.414897070 (604h22m15.285980115s) [noise=true , address=145.79.227.33:4904 , piece_id=KM563QF7HUNTIPTYGEPIM6DBCQSHZUYFLJQMGJ5DJDMWOWSKDB4A, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.415264499 (604h22m15.286347544s) [noise=true , address=77.90.31.12:4609 , piece_id=2SNAKTN72NMAYOAM5BNTNO6A6RD3XFIRNVVALVZLFRZIU47VEMXA, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.416022122 (604h22m15.287105167s) [noise=true , address=145.79.227.40:4068 , piece_id=LY4JRNBNZBFHSKINNAGVFYVS2JXSOO6ELNKUPQMNPVY5LWZ7A4EQ, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.418479571 (604h22m15.289562606s) [noise=true , address=145.79.227.52:4462 , piece_id=ENQYW6ECH5WXOZROCR6AZGFFA6JJ57MMRP62EQU6OFH2YODPFXAA, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:16.845346549 (604h22m16.716429584s) [upload=529718]: Next returned nil segment Dec 30 19:51:16.845364172 (604h22m16.716447207s) [upload=529718]: Waiting for error group managing segments... Dec 30 19:51:17.267522091 (604h22m17.138605126s) [noise=true , address=145.79.227.22:4532 , piece_id=34CFPGEIUKJAWIOHGCYT2IPARDF2Q6V4AZ75OEYJKWSGJP4XBLAA, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:17.381685254 (604h22m17.252768299s) [noise=true , address=145.79.227.14:4857 , piece_id=UAJPWWOJNMKWIR664WTJJTNLW6BSBWAXMBTM2KP27ZNDX5XUV7HA, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:17.491278576 (604h22m17.362361622s) [noise=true , address=77.90.31.16:4755 , piece_id=HSEFLVXWFD73XZ6FVA6L33OIZ4XMUD6KIC7OMUV2TZKSDJHBZIMQ, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:17.498649793 (604h22m17.369732838s) [noise=true , address=145.79.227.52:4462 , piece_id=ENQYW6ECH5WXOZROCR6AZGFFA6JJ57MMRP62EQU6OFH2YODPFXAA, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:17.508685632 (604h22m17.379768678s) [noise=true , address=77.90.31.12:4609 , piece_id=2SNAKTN72NMAYOAM5BNTNO6A6RD3XFIRNVVALVZLFRZIU47VEMXA, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:17.565292950 (604h22m17.436375986s) [noise=true , address=77.90.31.15:4710 , piece_id=HQYZ7QDCENX5OYIE4FJTF6VTC5RNQZPMH3KYEOBMYMB7EP2D3V2Q, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:17.587634352 (604h22m17.458717397s) [noise=true , address=45.91.116.8:4073 , piece_id=EV3LO3HZRKQF4A5R3HSPNCXKRE6BYLIQ2BQW6R5LMR4ROQQ6X3OA, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:17.655638912 (604h22m17.526721957s) [noise=true , address=45.91.116.7:4022 , piece_id=AMJKLUFDIY4M65Q4Y3DENPIQ2LP7IQDIBB5EJSXCMEE2MKPUOEDA, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:17.658922775 (604h22m17.53000581s) [noise=true , address=145.79.227.8:4359 , piece_id=XGBZZ64KTG2VJXGMOHZV5VY6STY2RJFFHANDSDXVY7UZ2DVWX3VA, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:17.678299712 (604h22m17.549382757s) [noise=true , address=45.91.116.8:4049 , piece_id=CDS23XTX2ILZKX7CWONNQ34F7W6G4BQHZOZLKFCWYJQKM3COPF4A, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:18.127069203 (604h22m17.998152249s) [noise=true , address=145.79.227.34:4937 , piece_id=GRDKPX53C5IPFCL2SE6L3NFCJRXUC7FPJ32HTFCCQGIIRUIITJ6Q, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:18.137212708 (604h22m18.008295743s) [noise=true , address=145.79.227.33:4904 , piece_id=KM563QF7HUNTIPTYGEPIM6DBCQSHZUYFLJQMGJ5DJDMWOWSKDB4A, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:18.160684672 (604h22m18.031767707s) [noise=true , address=145.79.227.40:4068 , piece_id=LY4JRNBNZBFHSKINNAGVFYVS2JXSOO6ELNKUPQMNPVY5LWZ7A4EQ, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:18.219837768 (604h22m18.090920833s) [noise=true , address=145.79.227.35:4487 , piece_id=YHVVDBISXCSLG3UW5XT7QEAKXOHSPS7RUY6HKZZS7NITA7OV47TQ, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:18.219866553 (604h22m18.090949598s) [seg_pos={0 0}, upload=529718]: Segment reached optimal threshold of 14 pieces. Dec 30 19:51:18.220084728 (604h22m18.091167763s) [noise=true , address=145.79.227.39:4040 , piece_id=OYYRO2KDG43DP2XYKFHWLSUJMSKQ6VC57GSW2PRNIA42XUTHEC2A, seg_pos={0 0}, upload=529718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gozJwRZvMK7Vbsdy775PtGynixvh3z5MHWwaLxzkrExseZosn): protocol: write tcp 188.166.225.170:44478->145.79.227.39:4040: use of closed network connection; write tcp 188.166.225.170:44478->145.79.227.39:4040: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44478->145.79.227.39:4040: use of closed network connection Dec 30 19:51:18.220092643 (604h22m18.091175678s) [noise=true , address=145.79.227.39:4040 , piece_id=OYYRO2KDG43DP2XYKFHWLSUJMSKQ6VC57GSW2PRNIA42XUTHEC2A, seg_pos={0 0}, upload=529718]: Long tail cancellation due to optimal threshold Dec 30 19:51:18.220170322 (604h22m18.091253357s) [noise=true , address=193.149.164.143:4704 , piece_id=LIPJT75FNGVN73FY277IE5FH6PXESYJXBCIUEVXJFQJ36QEHTH7A, seg_pos={0 0}, upload=529718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 30 19:51:18.220186161 (604h22m18.091269196s) [noise=true , address=193.149.164.143:4704 , piece_id=LIPJT75FNGVN73FY277IE5FH6PXESYJXBCIUEVXJFQJ36QEHTH7A, seg_pos={0 0}, upload=529718]: Long tail cancellation due to optimal threshold Dec 30 19:51:18.220218072 (604h22m18.091301107s) [noise=true , address=145.79.227.41:4099 , piece_id=IZN35ZJKWHB7DUAUFJ55HFUNJR5UMQ52YQN3CMYTTZUXMKJKN5UQ, seg_pos={0 0}, upload=529718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122dGFWNmsuheRmfEfNGWHftPpCcWv7HHXKymgJxsye6P7ZKhZT): protocol: write tcp 188.166.225.170:36572->145.79.227.41:4099: use of closed network connection; write tcp 188.166.225.170:36572->145.79.227.41:4099: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36572->145.79.227.41:4099: use of closed network connection Dec 30 19:51:18.220225947 (604h22m18.091308982s) [noise=true , address=145.79.227.41:4099 , piece_id=IZN35ZJKWHB7DUAUFJ55HFUNJR5UMQ52YQN3CMYTTZUXMKJKN5UQ, seg_pos={0 0}, upload=529718]: Long tail cancellation due to optimal threshold Dec 30 19:51:18.220496693 (604h22m18.091579728s) [noise=true , address=77.90.31.10:4526 , piece_id=EGGUMVBH572S2LZZWUYBKX2JSH2NVXC2BLHLXOKFUZKDMXHNPG4A, seg_pos={0 0}, upload=529718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: expected piece hash; context canceled; EOF Dec 30 19:51:18.220503786 (604h22m18.091586821s) [noise=true , address=77.90.31.10:4526 , piece_id=EGGUMVBH572S2LZZWUYBKX2JSH2NVXC2BLHLXOKFUZKDMXHNPG4A, seg_pos={0 0}, upload=529718]: Long tail cancellation due to optimal threshold Dec 30 19:51:18.220523634 (604h22m18.091606669s) [seg_pos={0 0}, upload=529718]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 19:51:18.220603907 (604h22m18.091686942s) [upload=529718]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:51:18.220611932 (604h22m18.091694967s) [upload=529718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:51:18.528318700 (604h22m18.399401735s) [upload=529718]: Done uploading segments... Dec 30 19:51:18.528354518 (604h22m18.399437553s) [upload=529718]: Upload finished. err: <nil> Duration: 15.402191678s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.....X..XX...X..X.....XX.X.....X.....X.................X...X] duration: 30.00258013s trace: https://jaeger.dev.storj.io/trace/62ea6ecd955e2692 FAIL (2025-12-30 19:54:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: WTJIIOZZJW5IXIFORC2BANV5S6GSZX3A6VCCMWCFTBDTCFAWB5VA) context deadline exceeded; piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: 72IJIO3E2WUUYDFDNBREZFSQVI5F772GM5X4XZZPQUWHOJLPK2QQ) context deadline exceeded ======LOGS====== 29761]: Starting upload Dec 30 19:54:00.971827755 (604h25m0.84291082s) [upload=529761]: Done starting upload Dec 30 19:54:00.972046472 (604h25m0.843129507s) [upload=529761]: Uploading segments... Dec 30 19:54:00.972095655 (604h25m0.843178711s) [upload=529761]: Got next segment. Inline: false Dec 30 19:54:00.972126975 (604h25m0.84321001s) [upload=529761]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 19:54:00.972132165 (604h25m0.84321524s) [upload=529761]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 19:54:01.217503947 (604h25m1.088586992s) [seg_pos={0 0}, upload=529761]: Begin upload segment... Dec 30 19:54:01.217570885 (604h25m1.08865393s) [seg_pos={0 0}, upload=529761]: Done begin upload segment. Dec 30 19:54:01.217605882 (604h25m1.088688937s) [noise=true , address=145.79.227.20:4437 , piece_id=ALTKEV25PB4XQGW7PYPGP7LXOBOJTBHE75VF6UKZLZA7NBIZRGKA, seg_pos={0 0}, upload=529761]: Uploading piece... Dec 30 19:54:01.217636811 (604h25m1.088719856s) [noise=true , address=45.91.116.5:4300 , piece_id=TC24DWZBNLBC3ZHABNXSN23NMVYCHJ3VHM3W7FSODQVKL4RLPJ6A, seg_pos={0 0}, upload=529761]: Uploading piece... Dec 30 19:54:01.217692477 (604h25m1.088775522s) [noise=true , address=145.79.227.17:4971 , piece_id=BBYHYMT6H2SI6XJIO3KO7PBX6EK46DUA4CAMM2VWPC57STOZLGFA, seg_pos={0 0}, upload=529761]: Uploading piece... Dec 30 19:54:01.218460690 (604h25m1.089543735s) [noise=true , address=77.90.31.17:4773 , piece_id=IV3VOWAQBVKIQ7MWYPE7PDZTEEARX555G6RMC7XJSNCU7XCKKCFQ, seg_pos={0 0}, upload=529761]: Uploading piece... Dec 30 19:54:01.218728860 (604h25m1.089811905s) [noise=true , address=45.91.116.9:4263 , piece_id=S77ZC7VREF5IF24QUXDI4SFYTOG4AVUHS76NMCL62KTM35OIBHBA, seg_pos={0 0}, upload=529761]: Uploading piece... Dec 30 19:54:01.218999104 (604h25m1.09008215s) [noise=true , address=145.79.227.39:4014 , piece_id=HFT4NOZ5RIEUTC3DLCQR3Z6JUGUGUVH4BVR3XYDDWUAL432MYP4A, seg_pos={0 0}, upload=529761]: Uploading piece... Dec 30 19:54:01.219110657 (604h25m1.090193692s) [noise=true , address=45.91.116.12:4226 , piece_id=Y4IAQFVBUR6JXLGWLXHT7DKQTJGXY36DKEKCPRKBMIWWKZ4NL6DA, seg_pos={0 0}, upload=529761]: Uploading piece... Dec 30 19:54:01.219549753 (604h25m1.090632798s) [noise=true , address=77.90.31.10:4520 , piece_id=WTJIIOZZJW5IXIFORC2BANV5S6GSZX3A6VCCMWCFTBDTCFAWB5VA, seg_pos={0 0}, upload=529761]: Uploading piece... Dec 30 19:54:01.219933063 (604h25m1.091016108s) [noise=true , address=145.79.227.15:4876 , piece_id=XABRDRBTOTGCBUUZDPJ7I35ZTSE33ATMFUYWKXB3EAAIAK235BKA, seg_pos={0 0}, upload=529761]: Uploading piece... Dec 30 19:54:01.220144646 (604h25m1.091227691s) [noise=true , address=145.79.227.36:4517 , piece_id=FNKAM5YZKOGHRUUP63V4IQ53OSIIORPCWJ4UOWSTD3DBAPYV3BHA, seg_pos={0 0}, upload=529761]: Uploading piece... Dec 30 19:54:01.220322094 (604h25m1.091405139s) [noise=true , address=77.90.31.15:4700 , piece_id=VMCFO2PPX77NLE6E3KX434M2BK26RBBGMUG775HF2MJZS2GOH2WQ, seg_pos={0 0}, upload=529761]: Uploading piece... Dec 30 19:54:01.220824331 (604h25m1.091907366s) [noise=true , address=45.91.116.11:4118 , piece_id=YD4RQAOP33CDDPNJKGW6AB3X47VXKYJBIPBI4DRVMQ2XGTEJSHYA, seg_pos={0 0}, upload=529761]: Uploading piece... Dec 30 19:54:01.221399044 (604h25m1.09248209s) [noise=true , address=145.79.227.23:4547 , piece_id=6G7YQF2SWESD3RWSEUZAG5RCCWJKQVZD4X4XUXYJHDZYZ4H6RVYQ, seg_pos={0 0}, upload=529761]: Uploading piece... Dec 30 19:54:01.221996632 (604h25m1.093079677s) [noise=true , address=145.79.227.51:4411 , piece_id=DDNNYRQ24RVYFXM7FIYCFWHOCEMAX66NVPJ4SWDQQ7WKBLU54QFA, seg_pos={0 0}, upload=529761]: Uploading piece... Dec 30 19:54:01.222447932 (604h25m1.093530977s) [noise=true , address=45.91.116.13:4155 , piece_id=72IJIO3E2WUUYDFDNBREZFSQVI5F772GM5X4XZZPQUWHOJLPK2QQ, seg_pos={0 0}, upload=529761]: Uploading piece... Dec 30 19:54:01.222795834 (604h25m1.093878879s) [noise=true , address=77.90.31.11:4555 , piece_id=4KQCJHSS3ED4NL2QIO7XG7VX5P3NBKO3JWFT565XBCNKKCVRHQOA, seg_pos={0 0}, upload=529761]: Uploading piece... Dec 30 19:54:02.814372916 (604h25m2.685455961s) [upload=529761]: Next returned nil segment Dec 30 19:54:02.814386151 (604h25m2.685469186s) [upload=529761]: Waiting for error group managing segments... Dec 30 19:54:03.191559262 (604h25m3.062642307s) [noise=true , address=145.79.227.17:4971 , piece_id=BBYHYMT6H2SI6XJIO3KO7PBX6EK46DUA4CAMM2VWPC57STOZLGFA, seg_pos={0 0}, upload=529761]: Done uploading piece. err: <nil> Dec 30 19:54:03.196306852 (604h25m3.067389897s) [noise=true , address=145.79.227.23:4547 , piece_id=6G7YQF2SWESD3RWSEUZAG5RCCWJKQVZD4X4XUXYJHDZYZ4H6RVYQ, seg_pos={0 0}, upload=529761]: Done uploading piece. err: <nil> Dec 30 19:54:03.197469086 (604h25m3.068552121s) [noise=true , address=145.79.227.51:4411 , piece_id=DDNNYRQ24RVYFXM7FIYCFWHOCEMAX66NVPJ4SWDQQ7WKBLU54QFA, seg_pos={0 0}, upload=529761]: Done uploading piece. err: <nil> Dec 30 19:54:03.212497021 (604h25m3.083580056s) [noise=true , address=145.79.227.20:4437 , piece_id=ALTKEV25PB4XQGW7PYPGP7LXOBOJTBHE75VF6UKZLZA7NBIZRGKA, seg_pos={0 0}, upload=529761]: Done uploading piece. err: <nil> Dec 30 19:54:03.212905800 (604h25m3.083988845s) [noise=true , address=145.79.227.15:4876 , piece_id=XABRDRBTOTGCBUUZDPJ7I35ZTSE33ATMFUYWKXB3EAAIAK235BKA, seg_pos={0 0}, upload=529761]: Done uploading piece. err: <nil> Dec 30 19:54:03.213920081 (604h25m3.085003116s) [noise=true , address=145.79.227.39:4014 , piece_id=HFT4NOZ5RIEUTC3DLCQR3Z6JUGUGUVH4BVR3XYDDWUAL432MYP4A, seg_pos={0 0}, upload=529761]: Done uploading piece. err: <nil> Dec 30 19:54:03.227126788 (604h25m3.098209833s) [noise=true , address=145.79.227.36:4517 , piece_id=FNKAM5YZKOGHRUUP63V4IQ53OSIIORPCWJ4UOWSTD3DBAPYV3BHA, seg_pos={0 0}, upload=529761]: Done uploading piece. err: <nil> Dec 30 19:54:03.396500439 (604h25m3.267583485s) [noise=true , address=77.90.31.10:4520 , piece_id=WTJIIOZZJW5IXIFORC2BANV5S6GSZX3A6VCCMWCFTBDTCFAWB5VA, seg_pos={0 0}, upload=529761]: Done uploading piece. err: <nil> Dec 30 19:54:03.516423741 (604h25m3.387506776s) [noise=true , address=77.90.31.15:4700 , piece_id=VMCFO2PPX77NLE6E3KX434M2BK26RBBGMUG775HF2MJZS2GOH2WQ, seg_pos={0 0}, upload=529761]: Done uploading piece. err: <nil> Dec 30 19:54:03.548322150 (604h25m3.419405195s) [noise=true , address=45.91.116.9:4263 , piece_id=S77ZC7VREF5IF24QUXDI4SFYTOG4AVUHS76NMCL62KTM35OIBHBA, seg_pos={0 0}, upload=529761]: Done uploading piece. err: <nil> Dec 30 19:54:03.578983784 (604h25m3.450066829s) [noise=true , address=45.91.116.12:4226 , piece_id=Y4IAQFVBUR6JXLGWLXHT7DKQTJGXY36DKEKCPRKBMIWWKZ4NL6DA, seg_pos={0 0}, upload=529761]: Done uploading piece. err: <nil> Dec 30 19:54:03.658414372 (604h25m3.529497417s) [noise=true , address=45.91.116.13:4155 , piece_id=72IJIO3E2WUUYDFDNBREZFSQVI5F772GM5X4XZZPQUWHOJLPK2QQ, seg_pos={0 0}, upload=529761]: Done uploading piece. err: <nil> Dec 30 19:54:03.672846182 (604h25m3.543929227s) [noise=true , address=45.91.116.5:4300 , piece_id=TC24DWZBNLBC3ZHABNXSN23NMVYCHJ3VHM3W7FSODQVKL4RLPJ6A, seg_pos={0 0}, upload=529761]: Done uploading piece. err: <nil> Dec 30 19:54:03.672871320 (604h25m3.543954365s) [seg_pos={0 0}, upload=529761]: Segment reached optimal threshold of 13 pieces. Dec 30 19:54:03.673113350 (604h25m3.544196395s) [noise=true , address=45.91.116.11:4118 , piece_id=YD4RQAOP33CDDPNJKGW6AB3X47VXKYJBIPBI4DRVMQ2XGTEJSHYA, seg_pos={0 0}, upload=529761]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:54692->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:54692->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54692->45.91.116.11:4118: use of closed network connection Dec 30 19:54:03.673121727 (604h25m3.544204772s) [noise=true , address=45.91.116.11:4118 , piece_id=YD4RQAOP33CDDPNJKGW6AB3X47VXKYJBIPBI4DRVMQ2XGTEJSHYA, seg_pos={0 0}, upload=529761]: Long tail cancellation due to optimal threshold Dec 30 19:54:03.673344661 (604h25m3.544427706s) [noise=true , address=77.90.31.11:4555 , piece_id=4KQCJHSS3ED4NL2QIO7XG7VX5P3NBKO3JWFT565XBCNKKCVRHQOA, seg_pos={0 0}, upload=529761]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 30 19:54:03.673361774 (604h25m3.544444819s) [noise=true , address=77.90.31.11:4555 , piece_id=4KQCJHSS3ED4NL2QIO7XG7VX5P3NBKO3JWFT565XBCNKKCVRHQOA, seg_pos={0 0}, upload=529761]: Long tail cancellation due to optimal threshold Dec 30 19:54:03.673459760 (604h25m3.544542795s) [noise=true , address=77.90.31.17:4773 , piece_id=IV3VOWAQBVKIQ7MWYPE7PDZTEEARX555G6RMC7XJSNCU7XCKKCFQ, seg_pos={0 0}, upload=529761]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: read tcp 188.166.225.170:41572->77.90.31.17:4773: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 19:54:03.673469980 (604h25m3.544553025s) [noise=true , address=77.90.31.17:4773 , piece_id=IV3VOWAQBVKIQ7MWYPE7PDZTEEARX555G6RMC7XJSNCU7XCKKCFQ, seg_pos={0 0}, upload=529761]: Long tail cancellation due to optimal threshold Dec 30 19:54:03.673494726 (604h25m3.544577771s) [seg_pos={0 0}, upload=529761]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 19:54:03.673583836 (604h25m3.544666882s) [upload=529761]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:54:03.673590399 (604h25m3.544673444s) [upload=529761]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:54:03.978363310 (604h25m3.849446355s) [upload=529761]: Done uploading segments... Dec 30 19:54:03.978384961 (604h25m3.849468007s) [upload=529761]: Upload finished. err: <nil> Duration: 30.002592564s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X..X.....................X......................X...] duration: 18.569320883s trace: https://jaeger.dev.storj.io/trace/fd3a56936ae71f2 FAIL (2025-12-30 19:53:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== piece_id=AMJKLUFDIY4M65Q4Y3DENPIQ2LP7IQDIBB5EJSXCMEE2MKPUOEDA, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.407338347 (604h22m15.278421392s) [noise=true , address=77.90.31.10:4526 , piece_id=EGGUMVBH572S2LZZWUYBKX2JSH2NVXC2BLHLXOKFUZKDMXHNPG4A, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.407626666 (604h22m15.278709711s) [noise=true , address=145.79.227.35:4487 , piece_id=YHVVDBISXCSLG3UW5XT7QEAKXOHSPS7RUY6HKZZS7NITA7OV47TQ, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.407809223 (604h22m15.278892258s) [noise=true , address=145.79.227.39:4040 , piece_id=OYYRO2KDG43DP2XYKFHWLSUJMSKQ6VC57GSW2PRNIA42XUTHEC2A, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.408000178 (604h22m15.279083213s) [noise=true , address=145.79.227.22:4532 , piece_id=34CFPGEIUKJAWIOHGCYT2IPARDF2Q6V4AZ75OEYJKWSGJP4XBLAA, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.408215778 (604h22m15.279298813s) [noise=true , address=77.90.31.16:4755 , piece_id=HSEFLVXWFD73XZ6FVA6L33OIZ4XMUD6KIC7OMUV2TZKSDJHBZIMQ, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.408338902 (604h22m15.279421937s) [noise=true , address=145.79.227.14:4857 , piece_id=UAJPWWOJNMKWIR664WTJJTNLW6BSBWAXMBTM2KP27ZNDX5XUV7HA, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.409170024 (604h22m15.28025306s) [noise=true , address=193.149.164.143:4704 , piece_id=LIPJT75FNGVN73FY277IE5FH6PXESYJXBCIUEVXJFQJ36QEHTH7A, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.410288113 (604h22m15.281371148s) [noise=true , address=145.79.227.34:4937 , piece_id=GRDKPX53C5IPFCL2SE6L3NFCJRXUC7FPJ32HTFCCQGIIRUIITJ6Q, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.411612275 (604h22m15.28269531s) [noise=true , address=45.91.116.8:4049 , piece_id=CDS23XTX2ILZKX7CWONNQ34F7W6G4BQHZOZLKFCWYJQKM3COPF4A, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.413868601 (604h22m15.284951646s) [noise=true , address=45.91.116.8:4073 , piece_id=EV3LO3HZRKQF4A5R3HSPNCXKRE6BYLIQ2BQW6R5LMR4ROQQ6X3OA, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.414897070 (604h22m15.285980115s) [noise=true , address=145.79.227.33:4904 , piece_id=KM563QF7HUNTIPTYGEPIM6DBCQSHZUYFLJQMGJ5DJDMWOWSKDB4A, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.415264499 (604h22m15.286347544s) [noise=true , address=77.90.31.12:4609 , piece_id=2SNAKTN72NMAYOAM5BNTNO6A6RD3XFIRNVVALVZLFRZIU47VEMXA, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.416022122 (604h22m15.287105167s) [noise=true , address=145.79.227.40:4068 , piece_id=LY4JRNBNZBFHSKINNAGVFYVS2JXSOO6ELNKUPQMNPVY5LWZ7A4EQ, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.418479571 (604h22m15.289562606s) [noise=true , address=145.79.227.52:4462 , piece_id=ENQYW6ECH5WXOZROCR6AZGFFA6JJ57MMRP62EQU6OFH2YODPFXAA, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:16.845346549 (604h22m16.716429584s) [upload=529718]: Next returned nil segment Dec 30 19:51:16.845364172 (604h22m16.716447207s) [upload=529718]: Waiting for error group managing segments... Dec 30 19:51:17.267522091 (604h22m17.138605126s) [noise=true , address=145.79.227.22:4532 , piece_id=34CFPGEIUKJAWIOHGCYT2IPARDF2Q6V4AZ75OEYJKWSGJP4XBLAA, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:17.381685254 (604h22m17.252768299s) [noise=true , address=145.79.227.14:4857 , piece_id=UAJPWWOJNMKWIR664WTJJTNLW6BSBWAXMBTM2KP27ZNDX5XUV7HA, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:17.491278576 (604h22m17.362361622s) [noise=true , address=77.90.31.16:4755 , piece_id=HSEFLVXWFD73XZ6FVA6L33OIZ4XMUD6KIC7OMUV2TZKSDJHBZIMQ, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:17.498649793 (604h22m17.369732838s) [noise=true , address=145.79.227.52:4462 , piece_id=ENQYW6ECH5WXOZROCR6AZGFFA6JJ57MMRP62EQU6OFH2YODPFXAA, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:17.508685632 (604h22m17.379768678s) [noise=true , address=77.90.31.12:4609 , piece_id=2SNAKTN72NMAYOAM5BNTNO6A6RD3XFIRNVVALVZLFRZIU47VEMXA, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:17.565292950 (604h22m17.436375986s) [noise=true , address=77.90.31.15:4710 , piece_id=HQYZ7QDCENX5OYIE4FJTF6VTC5RNQZPMH3KYEOBMYMB7EP2D3V2Q, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:17.587634352 (604h22m17.458717397s) [noise=true , address=45.91.116.8:4073 , piece_id=EV3LO3HZRKQF4A5R3HSPNCXKRE6BYLIQ2BQW6R5LMR4ROQQ6X3OA, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:17.655638912 (604h22m17.526721957s) [noise=true , address=45.91.116.7:4022 , piece_id=AMJKLUFDIY4M65Q4Y3DENPIQ2LP7IQDIBB5EJSXCMEE2MKPUOEDA, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:17.658922775 (604h22m17.53000581s) [noise=true , address=145.79.227.8:4359 , piece_id=XGBZZ64KTG2VJXGMOHZV5VY6STY2RJFFHANDSDXVY7UZ2DVWX3VA, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:17.678299712 (604h22m17.549382757s) [noise=true , address=45.91.116.8:4049 , piece_id=CDS23XTX2ILZKX7CWONNQ34F7W6G4BQHZOZLKFCWYJQKM3COPF4A, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:18.127069203 (604h22m17.998152249s) [noise=true , address=145.79.227.34:4937 , piece_id=GRDKPX53C5IPFCL2SE6L3NFCJRXUC7FPJ32HTFCCQGIIRUIITJ6Q, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:18.137212708 (604h22m18.008295743s) [noise=true , address=145.79.227.33:4904 , piece_id=KM563QF7HUNTIPTYGEPIM6DBCQSHZUYFLJQMGJ5DJDMWOWSKDB4A, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:18.160684672 (604h22m18.031767707s) [noise=true , address=145.79.227.40:4068 , piece_id=LY4JRNBNZBFHSKINNAGVFYVS2JXSOO6ELNKUPQMNPVY5LWZ7A4EQ, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:18.219837768 (604h22m18.090920833s) [noise=true , address=145.79.227.35:4487 , piece_id=YHVVDBISXCSLG3UW5XT7QEAKXOHSPS7RUY6HKZZS7NITA7OV47TQ, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:18.219866553 (604h22m18.090949598s) [seg_pos={0 0}, upload=529718]: Segment reached optimal threshold of 14 pieces. Dec 30 19:51:18.220084728 (604h22m18.091167763s) [noise=true , address=145.79.227.39:4040 , piece_id=OYYRO2KDG43DP2XYKFHWLSUJMSKQ6VC57GSW2PRNIA42XUTHEC2A, seg_pos={0 0}, upload=529718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gozJwRZvMK7Vbsdy775PtGynixvh3z5MHWwaLxzkrExseZosn): protocol: write tcp 188.166.225.170:44478->145.79.227.39:4040: use of closed network connection; write tcp 188.166.225.170:44478->145.79.227.39:4040: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44478->145.79.227.39:4040: use of closed network connection Dec 30 19:51:18.220092643 (604h22m18.091175678s) [noise=true , address=145.79.227.39:4040 , piece_id=OYYRO2KDG43DP2XYKFHWLSUJMSKQ6VC57GSW2PRNIA42XUTHEC2A, seg_pos={0 0}, upload=529718]: Long tail cancellation due to optimal threshold Dec 30 19:51:18.220170322 (604h22m18.091253357s) [noise=true , address=193.149.164.143:4704 , piece_id=LIPJT75FNGVN73FY277IE5FH6PXESYJXBCIUEVXJFQJ36QEHTH7A, seg_pos={0 0}, upload=529718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 30 19:51:18.220186161 (604h22m18.091269196s) [noise=true , address=193.149.164.143:4704 , piece_id=LIPJT75FNGVN73FY277IE5FH6PXESYJXBCIUEVXJFQJ36QEHTH7A, seg_pos={0 0}, upload=529718]: Long tail cancellation due to optimal threshold Dec 30 19:51:18.220218072 (604h22m18.091301107s) [noise=true , address=145.79.227.41:4099 , piece_id=IZN35ZJKWHB7DUAUFJ55HFUNJR5UMQ52YQN3CMYTTZUXMKJKN5UQ, seg_pos={0 0}, upload=529718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122dGFWNmsuheRmfEfNGWHftPpCcWv7HHXKymgJxsye6P7ZKhZT): protocol: write tcp 188.166.225.170:36572->145.79.227.41:4099: use of closed network connection; write tcp 188.166.225.170:36572->145.79.227.41:4099: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36572->145.79.227.41:4099: use of closed network connection Dec 30 19:51:18.220225947 (604h22m18.091308982s) [noise=true , address=145.79.227.41:4099 , piece_id=IZN35ZJKWHB7DUAUFJ55HFUNJR5UMQ52YQN3CMYTTZUXMKJKN5UQ, seg_pos={0 0}, upload=529718]: Long tail cancellation due to optimal threshold Dec 30 19:51:18.220496693 (604h22m18.091579728s) [noise=true , address=77.90.31.10:4526 , piece_id=EGGUMVBH572S2LZZWUYBKX2JSH2NVXC2BLHLXOKFUZKDMXHNPG4A, seg_pos={0 0}, upload=529718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: expected piece hash; context canceled; EOF Dec 30 19:51:18.220503786 (604h22m18.091586821s) [noise=true , address=77.90.31.10:4526 , piece_id=EGGUMVBH572S2LZZWUYBKX2JSH2NVXC2BLHLXOKFUZKDMXHNPG4A, seg_pos={0 0}, upload=529718]: Long tail cancellation due to optimal threshold Dec 30 19:51:18.220523634 (604h22m18.091606669s) [seg_pos={0 0}, upload=529718]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 19:51:18.220603907 (604h22m18.091686942s) [upload=529718]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:51:18.220611932 (604h22m18.091694967s) [upload=529718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:51:18.528318700 (604h22m18.399401735s) [upload=529718]: Done uploading segments... Dec 30 19:51:18.528354518 (604h22m18.399437553s) [upload=529718]: Upload finished. err: <nil> Duration: 18.56933936s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X.....X..XX...X..X.....XX.X.....X.....X.................X...] duration: 28.236148582s trace: https://jaeger.dev.storj.io/trace/77d763626a2de69f FAIL (2025-12-30 19:53:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== . Dec 30 19:50:49.706686384 (604h21m49.577769419s) [upload=529713]: Got next segment. Inline: false Dec 30 19:50:49.706728235 (604h21m49.57781127s) [upload=529713]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 19:50:49.706732713 (604h21m49.577815748s) [upload=529713]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 19:50:49.956439120 (604h21m49.827522155s) [seg_pos={0 0}, upload=529713]: Begin upload segment... Dec 30 19:50:49.956496509 (604h21m49.827579554s) [seg_pos={0 0}, upload=529713]: Done begin upload segment. Dec 30 19:50:49.956533911 (604h21m49.827616956s) [noise=true , address=145.79.227.17:4970 , piece_id=4JLGJNJ5NY6EKS263RE7Q4MZORF5TLLD25GSPVIOBYZDP2XEHVBA, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.956666412 (604h21m49.827749447s) [noise=true , address=145.79.227.51:4411 , piece_id=4FPWDJJUOXZJKPIADGCCDACHFPIECD6JTL6B3MRNX2767TD6AYXA, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.957306692 (604h21m49.828389727s) [noise=true , address=77.90.31.10:4517 , piece_id=N7PNEZJHTQIRH2H5JKUCTGYEUQNRUVAG7E3ZXSPLOSY4YB7NOPRA, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.957724868 (604h21m49.828807913s) [noise=true , address=145.79.227.39:4014 , piece_id=MFTOYKF3Q7TJO6GARDKQ4CZTJCDORXXB5WCNU67A27KMNLBFDCIA, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.957888850 (604h21m49.828971895s) [noise=true , address=145.79.227.11:4732 , piece_id=7A6NUULRBP4EKSCZ6K72XREWNXSCLEMKIEP5DMMQZBJBDFICEXUA, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.958315583 (604h21m49.829398628s) [noise=true , address=145.79.227.23:4546 , piece_id=EABR4DPGHN5R5JSGDYHOTFU5TUXDC3UVUVMJQI4TCXXFTMYYRT5Q, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.958450761 (604h21m49.829533796s) [noise=true , address=77.90.31.21:4878 , piece_id=AUE3ABLPLC4B6NR7TQ4X2RUB4LOWCQPMYVGBQYEZ2HFQUIN2ZJWA, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.958822308 (604h21m49.829905343s) [noise=true , address=77.90.31.16:4735 , piece_id=MO6I2KLJBXFRZ5BOUHUQF3ABPPMTRNPMOUYLXLWDG2XBEYSU6YGA, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.958932247 (604h21m49.830015292s) [noise=true , address=145.79.227.2:4113 , piece_id=7GKO34YXFFS3HXEG2YTAW6WXGNHQG5J7FD6IMEJIRIUHHLUBK7WQ, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.959417631 (604h21m49.830500676s) [noise=true , address=145.79.227.31:4799 , piece_id=LDIWHZNNXSWR4Y6EPQVNDFKABV2427WXOVDDOXAP3ABURMTZPK3A, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.959438411 (604h21m49.830521446s) [noise=true , address=45.91.116.10:4084 , piece_id=RUJXLLDM5FCYCGQEYOF2MI4EAYOD5LVQDRGZB2L2RMK5YQ5B6ZKQ, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.959901042 (604h21m49.830984087s) [noise=true , address=145.79.227.41:4085 , piece_id=RPIMMFDMQM24OFDLCQAWC2GWS5WKFFUFPNIAMIZKWZZS4RF54W7Q, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.960008917 (604h21m49.831091962s) [noise=true , address=45.91.116.13:4154 , piece_id=UHJB6FDP4BX7JEHZXKDDFMYVPHKTEUQI4WAHKMHIKSKEH6VYI42A, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.960389352 (604h21m49.831472397s) [noise=true , address=45.91.116.12:4226 , piece_id=HMHO3QMYPUCSBXCN2JILSDY5DVXCYXS25SYHSHB3EGHWVD4KNOUA, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.960565286 (604h21m49.831648332s) [noise=true , address=145.79.227.15:4873 , piece_id=7NH5E4SMR5JPZLAS7Q7PUCBHFE3AAOWGPGYTWVBT5IQBZBBTLTQQ, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.960897439 (604h21m49.831980474s) [noise=true , address=145.79.227.33:4870 , piece_id=RSW5PU23ZUPQGLTRM62SVGGLGVTIPLBSR5ZARGXJ3DVOC2UPM4EA, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:51.646909056 (604h21m51.517992101s) [upload=529713]: Next returned nil segment Dec 30 19:50:51.646928483 (604h21m51.518011529s) [upload=529713]: Waiting for error group managing segments... Dec 30 19:50:51.934292786 (604h21m51.805375821s) [noise=true , address=145.79.227.31:4799 , piece_id=LDIWHZNNXSWR4Y6EPQVNDFKABV2427WXOVDDOXAP3ABURMTZPK3A, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:51.940462203 (604h21m51.811545238s) [noise=true , address=145.79.227.51:4411 , piece_id=4FPWDJJUOXZJKPIADGCCDACHFPIECD6JTL6B3MRNX2767TD6AYXA, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:51.975314537 (604h21m51.846397572s) [noise=true , address=145.79.227.11:4732 , piece_id=7A6NUULRBP4EKSCZ6K72XREWNXSCLEMKIEP5DMMQZBJBDFICEXUA, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:51.986605326 (604h21m51.857688361s) [noise=true , address=145.79.227.17:4970 , piece_id=4JLGJNJ5NY6EKS263RE7Q4MZORF5TLLD25GSPVIOBYZDP2XEHVBA, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:52.026019757 (604h21m51.897102802s) [noise=true , address=145.79.227.41:4085 , piece_id=RPIMMFDMQM24OFDLCQAWC2GWS5WKFFUFPNIAMIZKWZZS4RF54W7Q, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:52.030867858 (604h21m51.901950893s) [noise=true , address=145.79.227.39:4014 , piece_id=MFTOYKF3Q7TJO6GARDKQ4CZTJCDORXXB5WCNU67A27KMNLBFDCIA, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:52.177525915 (604h21m52.04860896s) [noise=true , address=77.90.31.10:4517 , piece_id=N7PNEZJHTQIRH2H5JKUCTGYEUQNRUVAG7E3ZXSPLOSY4YB7NOPRA, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:52.202966258 (604h21m52.074049293s) [noise=true , address=77.90.31.16:4735 , piece_id=MO6I2KLJBXFRZ5BOUHUQF3ABPPMTRNPMOUYLXLWDG2XBEYSU6YGA, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:52.219992708 (604h21m52.091075743s) [noise=true , address=77.90.31.21:4878 , piece_id=AUE3ABLPLC4B6NR7TQ4X2RUB4LOWCQPMYVGBQYEZ2HFQUIN2ZJWA, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:52.230964761 (604h21m52.102047796s) [noise=true , address=145.79.227.15:4873 , piece_id=7NH5E4SMR5JPZLAS7Q7PUCBHFE3AAOWGPGYTWVBT5IQBZBBTLTQQ, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:52.274453189 (604h21m52.145536234s) [noise=true , address=45.91.116.12:4226 , piece_id=HMHO3QMYPUCSBXCN2JILSDY5DVXCYXS25SYHSHB3EGHWVD4KNOUA, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:52.302329220 (604h21m52.173412255s) [noise=true , address=45.91.116.13:4154 , piece_id=UHJB6FDP4BX7JEHZXKDDFMYVPHKTEUQI4WAHKMHIKSKEH6VYI42A, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:52.364019437 (604h21m52.235102482s) [noise=true , address=45.91.116.10:4084 , piece_id=RUJXLLDM5FCYCGQEYOF2MI4EAYOD5LVQDRGZB2L2RMK5YQ5B6ZKQ, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:52.364038934 (604h21m52.235121969s) [seg_pos={0 0}, upload=529713]: Segment reached optimal threshold of 13 pieces. Dec 30 19:50:52.364227393 (604h21m52.235310438s) [noise=true , address=145.79.227.23:4546 , piece_id=EABR4DPGHN5R5JSGDYHOTFU5TUXDC3UVUVMJQI4TCXXFTMYYRT5Q, seg_pos={0 0}, upload=529713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: write tcp 188.166.225.170:51884->145.79.227.23:4546: use of closed network connection; write tcp 188.166.225.170:51884->145.79.227.23:4546: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51884->145.79.227.23:4546: use of closed network connection Dec 30 19:50:52.364235579 (604h21m52.235318614s) [noise=true , address=145.79.227.23:4546 , piece_id=EABR4DPGHN5R5JSGDYHOTFU5TUXDC3UVUVMJQI4TCXXFTMYYRT5Q, seg_pos={0 0}, upload=529713]: Long tail cancellation due to optimal threshold Dec 30 19:50:52.364465657 (604h21m52.235548692s) [noise=true , address=145.79.227.2:4113 , piece_id=7GKO34YXFFS3HXEG2YTAW6WXGNHQG5J7FD6IMEJIRIUHHLUBK7WQ, seg_pos={0 0}, upload=529713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:38802->145.79.227.2:4113: use of closed network connection; write tcp 188.166.225.170:38802->145.79.227.2:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38802->145.79.227.2:4113: use of closed network connection Dec 30 19:50:52.364474493 (604h21m52.235557538s) [noise=true , address=145.79.227.2:4113 , piece_id=7GKO34YXFFS3HXEG2YTAW6WXGNHQG5J7FD6IMEJIRIUHHLUBK7WQ, seg_pos={0 0}, upload=529713]: Long tail cancellation due to optimal threshold Dec 30 19:50:52.364541711 (604h21m52.235624746s) [noise=true , address=145.79.227.33:4870 , piece_id=RSW5PU23ZUPQGLTRM62SVGGLGVTIPLBSR5ZARGXJ3DVOC2UPM4EA, seg_pos={0 0}, upload=529713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 19:50:52.364561259 (604h21m52.235644304s) [noise=true , address=145.79.227.33:4870 , piece_id=RSW5PU23ZUPQGLTRM62SVGGLGVTIPLBSR5ZARGXJ3DVOC2UPM4EA, seg_pos={0 0}, upload=529713]: Long tail cancellation due to optimal threshold Dec 30 19:50:52.364605673 (604h21m52.235688708s) [seg_pos={0 0}, upload=529713]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 19:50:52.364730571 (604h21m52.235813617s) [upload=529713]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:50:52.364736002 (604h21m52.235819047s) [upload=529713]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:50:52.655672157 (604h21m52.526755202s) [upload=529713]: Done uploading segments... Dec 30 19:50:52.655698627 (604h21m52.526781682s) [upload=529713]: Upload finished. err: <nil> Duration: 28.236176947s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X..X.....................X......................X..] duration: 39.474937005s trace: https://jaeger.dev.storj.io/trace/fd3a56936ae71f2 FAIL (2025-12-30 19:53:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== piece_id=AMJKLUFDIY4M65Q4Y3DENPIQ2LP7IQDIBB5EJSXCMEE2MKPUOEDA, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.407338347 (604h22m15.278421392s) [noise=true , address=77.90.31.10:4526 , piece_id=EGGUMVBH572S2LZZWUYBKX2JSH2NVXC2BLHLXOKFUZKDMXHNPG4A, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.407626666 (604h22m15.278709711s) [noise=true , address=145.79.227.35:4487 , piece_id=YHVVDBISXCSLG3UW5XT7QEAKXOHSPS7RUY6HKZZS7NITA7OV47TQ, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.407809223 (604h22m15.278892258s) [noise=true , address=145.79.227.39:4040 , piece_id=OYYRO2KDG43DP2XYKFHWLSUJMSKQ6VC57GSW2PRNIA42XUTHEC2A, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.408000178 (604h22m15.279083213s) [noise=true , address=145.79.227.22:4532 , piece_id=34CFPGEIUKJAWIOHGCYT2IPARDF2Q6V4AZ75OEYJKWSGJP4XBLAA, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.408215778 (604h22m15.279298813s) [noise=true , address=77.90.31.16:4755 , piece_id=HSEFLVXWFD73XZ6FVA6L33OIZ4XMUD6KIC7OMUV2TZKSDJHBZIMQ, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.408338902 (604h22m15.279421937s) [noise=true , address=145.79.227.14:4857 , piece_id=UAJPWWOJNMKWIR664WTJJTNLW6BSBWAXMBTM2KP27ZNDX5XUV7HA, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.409170024 (604h22m15.28025306s) [noise=true , address=193.149.164.143:4704 , piece_id=LIPJT75FNGVN73FY277IE5FH6PXESYJXBCIUEVXJFQJ36QEHTH7A, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.410288113 (604h22m15.281371148s) [noise=true , address=145.79.227.34:4937 , piece_id=GRDKPX53C5IPFCL2SE6L3NFCJRXUC7FPJ32HTFCCQGIIRUIITJ6Q, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.411612275 (604h22m15.28269531s) [noise=true , address=45.91.116.8:4049 , piece_id=CDS23XTX2ILZKX7CWONNQ34F7W6G4BQHZOZLKFCWYJQKM3COPF4A, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.413868601 (604h22m15.284951646s) [noise=true , address=45.91.116.8:4073 , piece_id=EV3LO3HZRKQF4A5R3HSPNCXKRE6BYLIQ2BQW6R5LMR4ROQQ6X3OA, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.414897070 (604h22m15.285980115s) [noise=true , address=145.79.227.33:4904 , piece_id=KM563QF7HUNTIPTYGEPIM6DBCQSHZUYFLJQMGJ5DJDMWOWSKDB4A, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.415264499 (604h22m15.286347544s) [noise=true , address=77.90.31.12:4609 , piece_id=2SNAKTN72NMAYOAM5BNTNO6A6RD3XFIRNVVALVZLFRZIU47VEMXA, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.416022122 (604h22m15.287105167s) [noise=true , address=145.79.227.40:4068 , piece_id=LY4JRNBNZBFHSKINNAGVFYVS2JXSOO6ELNKUPQMNPVY5LWZ7A4EQ, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.418479571 (604h22m15.289562606s) [noise=true , address=145.79.227.52:4462 , piece_id=ENQYW6ECH5WXOZROCR6AZGFFA6JJ57MMRP62EQU6OFH2YODPFXAA, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:16.845346549 (604h22m16.716429584s) [upload=529718]: Next returned nil segment Dec 30 19:51:16.845364172 (604h22m16.716447207s) [upload=529718]: Waiting for error group managing segments... Dec 30 19:51:17.267522091 (604h22m17.138605126s) [noise=true , address=145.79.227.22:4532 , piece_id=34CFPGEIUKJAWIOHGCYT2IPARDF2Q6V4AZ75OEYJKWSGJP4XBLAA, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:17.381685254 (604h22m17.252768299s) [noise=true , address=145.79.227.14:4857 , piece_id=UAJPWWOJNMKWIR664WTJJTNLW6BSBWAXMBTM2KP27ZNDX5XUV7HA, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:17.491278576 (604h22m17.362361622s) [noise=true , address=77.90.31.16:4755 , piece_id=HSEFLVXWFD73XZ6FVA6L33OIZ4XMUD6KIC7OMUV2TZKSDJHBZIMQ, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:17.498649793 (604h22m17.369732838s) [noise=true , address=145.79.227.52:4462 , piece_id=ENQYW6ECH5WXOZROCR6AZGFFA6JJ57MMRP62EQU6OFH2YODPFXAA, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:17.508685632 (604h22m17.379768678s) [noise=true , address=77.90.31.12:4609 , piece_id=2SNAKTN72NMAYOAM5BNTNO6A6RD3XFIRNVVALVZLFRZIU47VEMXA, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:17.565292950 (604h22m17.436375986s) [noise=true , address=77.90.31.15:4710 , piece_id=HQYZ7QDCENX5OYIE4FJTF6VTC5RNQZPMH3KYEOBMYMB7EP2D3V2Q, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:17.587634352 (604h22m17.458717397s) [noise=true , address=45.91.116.8:4073 , piece_id=EV3LO3HZRKQF4A5R3HSPNCXKRE6BYLIQ2BQW6R5LMR4ROQQ6X3OA, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:17.655638912 (604h22m17.526721957s) [noise=true , address=45.91.116.7:4022 , piece_id=AMJKLUFDIY4M65Q4Y3DENPIQ2LP7IQDIBB5EJSXCMEE2MKPUOEDA, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:17.658922775 (604h22m17.53000581s) [noise=true , address=145.79.227.8:4359 , piece_id=XGBZZ64KTG2VJXGMOHZV5VY6STY2RJFFHANDSDXVY7UZ2DVWX3VA, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:17.678299712 (604h22m17.549382757s) [noise=true , address=45.91.116.8:4049 , piece_id=CDS23XTX2ILZKX7CWONNQ34F7W6G4BQHZOZLKFCWYJQKM3COPF4A, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:18.127069203 (604h22m17.998152249s) [noise=true , address=145.79.227.34:4937 , piece_id=GRDKPX53C5IPFCL2SE6L3NFCJRXUC7FPJ32HTFCCQGIIRUIITJ6Q, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:18.137212708 (604h22m18.008295743s) [noise=true , address=145.79.227.33:4904 , piece_id=KM563QF7HUNTIPTYGEPIM6DBCQSHZUYFLJQMGJ5DJDMWOWSKDB4A, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:18.160684672 (604h22m18.031767707s) [noise=true , address=145.79.227.40:4068 , piece_id=LY4JRNBNZBFHSKINNAGVFYVS2JXSOO6ELNKUPQMNPVY5LWZ7A4EQ, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:18.219837768 (604h22m18.090920833s) [noise=true , address=145.79.227.35:4487 , piece_id=YHVVDBISXCSLG3UW5XT7QEAKXOHSPS7RUY6HKZZS7NITA7OV47TQ, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:18.219866553 (604h22m18.090949598s) [seg_pos={0 0}, upload=529718]: Segment reached optimal threshold of 14 pieces. Dec 30 19:51:18.220084728 (604h22m18.091167763s) [noise=true , address=145.79.227.39:4040 , piece_id=OYYRO2KDG43DP2XYKFHWLSUJMSKQ6VC57GSW2PRNIA42XUTHEC2A, seg_pos={0 0}, upload=529718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gozJwRZvMK7Vbsdy775PtGynixvh3z5MHWwaLxzkrExseZosn): protocol: write tcp 188.166.225.170:44478->145.79.227.39:4040: use of closed network connection; write tcp 188.166.225.170:44478->145.79.227.39:4040: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44478->145.79.227.39:4040: use of closed network connection Dec 30 19:51:18.220092643 (604h22m18.091175678s) [noise=true , address=145.79.227.39:4040 , piece_id=OYYRO2KDG43DP2XYKFHWLSUJMSKQ6VC57GSW2PRNIA42XUTHEC2A, seg_pos={0 0}, upload=529718]: Long tail cancellation due to optimal threshold Dec 30 19:51:18.220170322 (604h22m18.091253357s) [noise=true , address=193.149.164.143:4704 , piece_id=LIPJT75FNGVN73FY277IE5FH6PXESYJXBCIUEVXJFQJ36QEHTH7A, seg_pos={0 0}, upload=529718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 30 19:51:18.220186161 (604h22m18.091269196s) [noise=true , address=193.149.164.143:4704 , piece_id=LIPJT75FNGVN73FY277IE5FH6PXESYJXBCIUEVXJFQJ36QEHTH7A, seg_pos={0 0}, upload=529718]: Long tail cancellation due to optimal threshold Dec 30 19:51:18.220218072 (604h22m18.091301107s) [noise=true , address=145.79.227.41:4099 , piece_id=IZN35ZJKWHB7DUAUFJ55HFUNJR5UMQ52YQN3CMYTTZUXMKJKN5UQ, seg_pos={0 0}, upload=529718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122dGFWNmsuheRmfEfNGWHftPpCcWv7HHXKymgJxsye6P7ZKhZT): protocol: write tcp 188.166.225.170:36572->145.79.227.41:4099: use of closed network connection; write tcp 188.166.225.170:36572->145.79.227.41:4099: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36572->145.79.227.41:4099: use of closed network connection Dec 30 19:51:18.220225947 (604h22m18.091308982s) [noise=true , address=145.79.227.41:4099 , piece_id=IZN35ZJKWHB7DUAUFJ55HFUNJR5UMQ52YQN3CMYTTZUXMKJKN5UQ, seg_pos={0 0}, upload=529718]: Long tail cancellation due to optimal threshold Dec 30 19:51:18.220496693 (604h22m18.091579728s) [noise=true , address=77.90.31.10:4526 , piece_id=EGGUMVBH572S2LZZWUYBKX2JSH2NVXC2BLHLXOKFUZKDMXHNPG4A, seg_pos={0 0}, upload=529718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: expected piece hash; context canceled; EOF Dec 30 19:51:18.220503786 (604h22m18.091586821s) [noise=true , address=77.90.31.10:4526 , piece_id=EGGUMVBH572S2LZZWUYBKX2JSH2NVXC2BLHLXOKFUZKDMXHNPG4A, seg_pos={0 0}, upload=529718]: Long tail cancellation due to optimal threshold Dec 30 19:51:18.220523634 (604h22m18.091606669s) [seg_pos={0 0}, upload=529718]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 19:51:18.220603907 (604h22m18.091686942s) [upload=529718]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:51:18.220611932 (604h22m18.091694967s) [upload=529718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:51:18.528318700 (604h22m18.399401735s) [upload=529718]: Done uploading segments... Dec 30 19:51:18.528354518 (604h22m18.399437553s) [upload=529718]: Upload finished. err: <nil> Duration: 39.474971482s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.X.....X..XX...X..X.....XX.X.....X.....X.................X..] duration: 28.831284203s trace: https://jaeger.dev.storj.io/trace/77d763626a2de69f FAIL (2025-12-30 19:52:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== . Dec 30 19:50:49.706686384 (604h21m49.577769419s) [upload=529713]: Got next segment. Inline: false Dec 30 19:50:49.706728235 (604h21m49.57781127s) [upload=529713]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 19:50:49.706732713 (604h21m49.577815748s) [upload=529713]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 19:50:49.956439120 (604h21m49.827522155s) [seg_pos={0 0}, upload=529713]: Begin upload segment... Dec 30 19:50:49.956496509 (604h21m49.827579554s) [seg_pos={0 0}, upload=529713]: Done begin upload segment. Dec 30 19:50:49.956533911 (604h21m49.827616956s) [noise=true , address=145.79.227.17:4970 , piece_id=4JLGJNJ5NY6EKS263RE7Q4MZORF5TLLD25GSPVIOBYZDP2XEHVBA, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.956666412 (604h21m49.827749447s) [noise=true , address=145.79.227.51:4411 , piece_id=4FPWDJJUOXZJKPIADGCCDACHFPIECD6JTL6B3MRNX2767TD6AYXA, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.957306692 (604h21m49.828389727s) [noise=true , address=77.90.31.10:4517 , piece_id=N7PNEZJHTQIRH2H5JKUCTGYEUQNRUVAG7E3ZXSPLOSY4YB7NOPRA, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.957724868 (604h21m49.828807913s) [noise=true , address=145.79.227.39:4014 , piece_id=MFTOYKF3Q7TJO6GARDKQ4CZTJCDORXXB5WCNU67A27KMNLBFDCIA, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.957888850 (604h21m49.828971895s) [noise=true , address=145.79.227.11:4732 , piece_id=7A6NUULRBP4EKSCZ6K72XREWNXSCLEMKIEP5DMMQZBJBDFICEXUA, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.958315583 (604h21m49.829398628s) [noise=true , address=145.79.227.23:4546 , piece_id=EABR4DPGHN5R5JSGDYHOTFU5TUXDC3UVUVMJQI4TCXXFTMYYRT5Q, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.958450761 (604h21m49.829533796s) [noise=true , address=77.90.31.21:4878 , piece_id=AUE3ABLPLC4B6NR7TQ4X2RUB4LOWCQPMYVGBQYEZ2HFQUIN2ZJWA, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.958822308 (604h21m49.829905343s) [noise=true , address=77.90.31.16:4735 , piece_id=MO6I2KLJBXFRZ5BOUHUQF3ABPPMTRNPMOUYLXLWDG2XBEYSU6YGA, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.958932247 (604h21m49.830015292s) [noise=true , address=145.79.227.2:4113 , piece_id=7GKO34YXFFS3HXEG2YTAW6WXGNHQG5J7FD6IMEJIRIUHHLUBK7WQ, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.959417631 (604h21m49.830500676s) [noise=true , address=145.79.227.31:4799 , piece_id=LDIWHZNNXSWR4Y6EPQVNDFKABV2427WXOVDDOXAP3ABURMTZPK3A, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.959438411 (604h21m49.830521446s) [noise=true , address=45.91.116.10:4084 , piece_id=RUJXLLDM5FCYCGQEYOF2MI4EAYOD5LVQDRGZB2L2RMK5YQ5B6ZKQ, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.959901042 (604h21m49.830984087s) [noise=true , address=145.79.227.41:4085 , piece_id=RPIMMFDMQM24OFDLCQAWC2GWS5WKFFUFPNIAMIZKWZZS4RF54W7Q, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.960008917 (604h21m49.831091962s) [noise=true , address=45.91.116.13:4154 , piece_id=UHJB6FDP4BX7JEHZXKDDFMYVPHKTEUQI4WAHKMHIKSKEH6VYI42A, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.960389352 (604h21m49.831472397s) [noise=true , address=45.91.116.12:4226 , piece_id=HMHO3QMYPUCSBXCN2JILSDY5DVXCYXS25SYHSHB3EGHWVD4KNOUA, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.960565286 (604h21m49.831648332s) [noise=true , address=145.79.227.15:4873 , piece_id=7NH5E4SMR5JPZLAS7Q7PUCBHFE3AAOWGPGYTWVBT5IQBZBBTLTQQ, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.960897439 (604h21m49.831980474s) [noise=true , address=145.79.227.33:4870 , piece_id=RSW5PU23ZUPQGLTRM62SVGGLGVTIPLBSR5ZARGXJ3DVOC2UPM4EA, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:51.646909056 (604h21m51.517992101s) [upload=529713]: Next returned nil segment Dec 30 19:50:51.646928483 (604h21m51.518011529s) [upload=529713]: Waiting for error group managing segments... Dec 30 19:50:51.934292786 (604h21m51.805375821s) [noise=true , address=145.79.227.31:4799 , piece_id=LDIWHZNNXSWR4Y6EPQVNDFKABV2427WXOVDDOXAP3ABURMTZPK3A, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:51.940462203 (604h21m51.811545238s) [noise=true , address=145.79.227.51:4411 , piece_id=4FPWDJJUOXZJKPIADGCCDACHFPIECD6JTL6B3MRNX2767TD6AYXA, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:51.975314537 (604h21m51.846397572s) [noise=true , address=145.79.227.11:4732 , piece_id=7A6NUULRBP4EKSCZ6K72XREWNXSCLEMKIEP5DMMQZBJBDFICEXUA, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:51.986605326 (604h21m51.857688361s) [noise=true , address=145.79.227.17:4970 , piece_id=4JLGJNJ5NY6EKS263RE7Q4MZORF5TLLD25GSPVIOBYZDP2XEHVBA, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:52.026019757 (604h21m51.897102802s) [noise=true , address=145.79.227.41:4085 , piece_id=RPIMMFDMQM24OFDLCQAWC2GWS5WKFFUFPNIAMIZKWZZS4RF54W7Q, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:52.030867858 (604h21m51.901950893s) [noise=true , address=145.79.227.39:4014 , piece_id=MFTOYKF3Q7TJO6GARDKQ4CZTJCDORXXB5WCNU67A27KMNLBFDCIA, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:52.177525915 (604h21m52.04860896s) [noise=true , address=77.90.31.10:4517 , piece_id=N7PNEZJHTQIRH2H5JKUCTGYEUQNRUVAG7E3ZXSPLOSY4YB7NOPRA, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:52.202966258 (604h21m52.074049293s) [noise=true , address=77.90.31.16:4735 , piece_id=MO6I2KLJBXFRZ5BOUHUQF3ABPPMTRNPMOUYLXLWDG2XBEYSU6YGA, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:52.219992708 (604h21m52.091075743s) [noise=true , address=77.90.31.21:4878 , piece_id=AUE3ABLPLC4B6NR7TQ4X2RUB4LOWCQPMYVGBQYEZ2HFQUIN2ZJWA, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:52.230964761 (604h21m52.102047796s) [noise=true , address=145.79.227.15:4873 , piece_id=7NH5E4SMR5JPZLAS7Q7PUCBHFE3AAOWGPGYTWVBT5IQBZBBTLTQQ, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:52.274453189 (604h21m52.145536234s) [noise=true , address=45.91.116.12:4226 , piece_id=HMHO3QMYPUCSBXCN2JILSDY5DVXCYXS25SYHSHB3EGHWVD4KNOUA, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:52.302329220 (604h21m52.173412255s) [noise=true , address=45.91.116.13:4154 , piece_id=UHJB6FDP4BX7JEHZXKDDFMYVPHKTEUQI4WAHKMHIKSKEH6VYI42A, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:52.364019437 (604h21m52.235102482s) [noise=true , address=45.91.116.10:4084 , piece_id=RUJXLLDM5FCYCGQEYOF2MI4EAYOD5LVQDRGZB2L2RMK5YQ5B6ZKQ, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:52.364038934 (604h21m52.235121969s) [seg_pos={0 0}, upload=529713]: Segment reached optimal threshold of 13 pieces. Dec 30 19:50:52.364227393 (604h21m52.235310438s) [noise=true , address=145.79.227.23:4546 , piece_id=EABR4DPGHN5R5JSGDYHOTFU5TUXDC3UVUVMJQI4TCXXFTMYYRT5Q, seg_pos={0 0}, upload=529713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: write tcp 188.166.225.170:51884->145.79.227.23:4546: use of closed network connection; write tcp 188.166.225.170:51884->145.79.227.23:4546: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51884->145.79.227.23:4546: use of closed network connection Dec 30 19:50:52.364235579 (604h21m52.235318614s) [noise=true , address=145.79.227.23:4546 , piece_id=EABR4DPGHN5R5JSGDYHOTFU5TUXDC3UVUVMJQI4TCXXFTMYYRT5Q, seg_pos={0 0}, upload=529713]: Long tail cancellation due to optimal threshold Dec 30 19:50:52.364465657 (604h21m52.235548692s) [noise=true , address=145.79.227.2:4113 , piece_id=7GKO34YXFFS3HXEG2YTAW6WXGNHQG5J7FD6IMEJIRIUHHLUBK7WQ, seg_pos={0 0}, upload=529713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:38802->145.79.227.2:4113: use of closed network connection; write tcp 188.166.225.170:38802->145.79.227.2:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38802->145.79.227.2:4113: use of closed network connection Dec 30 19:50:52.364474493 (604h21m52.235557538s) [noise=true , address=145.79.227.2:4113 , piece_id=7GKO34YXFFS3HXEG2YTAW6WXGNHQG5J7FD6IMEJIRIUHHLUBK7WQ, seg_pos={0 0}, upload=529713]: Long tail cancellation due to optimal threshold Dec 30 19:50:52.364541711 (604h21m52.235624746s) [noise=true , address=145.79.227.33:4870 , piece_id=RSW5PU23ZUPQGLTRM62SVGGLGVTIPLBSR5ZARGXJ3DVOC2UPM4EA, seg_pos={0 0}, upload=529713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 19:50:52.364561259 (604h21m52.235644304s) [noise=true , address=145.79.227.33:4870 , piece_id=RSW5PU23ZUPQGLTRM62SVGGLGVTIPLBSR5ZARGXJ3DVOC2UPM4EA, seg_pos={0 0}, upload=529713]: Long tail cancellation due to optimal threshold Dec 30 19:50:52.364605673 (604h21m52.235688708s) [seg_pos={0 0}, upload=529713]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 19:50:52.364730571 (604h21m52.235813617s) [upload=529713]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:50:52.364736002 (604h21m52.235819047s) [upload=529713]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:50:52.655672157 (604h21m52.526755202s) [upload=529713]: Done uploading segments... Dec 30 19:50:52.655698627 (604h21m52.526781682s) [upload=529713]: Upload finished. err: <nil> Duration: 28.831316205s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X..X.....................X......................X.] duration: 24.192611579s trace: https://jaeger.dev.storj.io/trace/fd3a56936ae71f2 FAIL (2025-12-30 19:52:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== piece_id=AMJKLUFDIY4M65Q4Y3DENPIQ2LP7IQDIBB5EJSXCMEE2MKPUOEDA, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.407338347 (604h22m15.278421392s) [noise=true , address=77.90.31.10:4526 , piece_id=EGGUMVBH572S2LZZWUYBKX2JSH2NVXC2BLHLXOKFUZKDMXHNPG4A, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.407626666 (604h22m15.278709711s) [noise=true , address=145.79.227.35:4487 , piece_id=YHVVDBISXCSLG3UW5XT7QEAKXOHSPS7RUY6HKZZS7NITA7OV47TQ, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.407809223 (604h22m15.278892258s) [noise=true , address=145.79.227.39:4040 , piece_id=OYYRO2KDG43DP2XYKFHWLSUJMSKQ6VC57GSW2PRNIA42XUTHEC2A, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.408000178 (604h22m15.279083213s) [noise=true , address=145.79.227.22:4532 , piece_id=34CFPGEIUKJAWIOHGCYT2IPARDF2Q6V4AZ75OEYJKWSGJP4XBLAA, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.408215778 (604h22m15.279298813s) [noise=true , address=77.90.31.16:4755 , piece_id=HSEFLVXWFD73XZ6FVA6L33OIZ4XMUD6KIC7OMUV2TZKSDJHBZIMQ, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.408338902 (604h22m15.279421937s) [noise=true , address=145.79.227.14:4857 , piece_id=UAJPWWOJNMKWIR664WTJJTNLW6BSBWAXMBTM2KP27ZNDX5XUV7HA, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.409170024 (604h22m15.28025306s) [noise=true , address=193.149.164.143:4704 , piece_id=LIPJT75FNGVN73FY277IE5FH6PXESYJXBCIUEVXJFQJ36QEHTH7A, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.410288113 (604h22m15.281371148s) [noise=true , address=145.79.227.34:4937 , piece_id=GRDKPX53C5IPFCL2SE6L3NFCJRXUC7FPJ32HTFCCQGIIRUIITJ6Q, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.411612275 (604h22m15.28269531s) [noise=true , address=45.91.116.8:4049 , piece_id=CDS23XTX2ILZKX7CWONNQ34F7W6G4BQHZOZLKFCWYJQKM3COPF4A, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.413868601 (604h22m15.284951646s) [noise=true , address=45.91.116.8:4073 , piece_id=EV3LO3HZRKQF4A5R3HSPNCXKRE6BYLIQ2BQW6R5LMR4ROQQ6X3OA, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.414897070 (604h22m15.285980115s) [noise=true , address=145.79.227.33:4904 , piece_id=KM563QF7HUNTIPTYGEPIM6DBCQSHZUYFLJQMGJ5DJDMWOWSKDB4A, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.415264499 (604h22m15.286347544s) [noise=true , address=77.90.31.12:4609 , piece_id=2SNAKTN72NMAYOAM5BNTNO6A6RD3XFIRNVVALVZLFRZIU47VEMXA, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.416022122 (604h22m15.287105167s) [noise=true , address=145.79.227.40:4068 , piece_id=LY4JRNBNZBFHSKINNAGVFYVS2JXSOO6ELNKUPQMNPVY5LWZ7A4EQ, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.418479571 (604h22m15.289562606s) [noise=true , address=145.79.227.52:4462 , piece_id=ENQYW6ECH5WXOZROCR6AZGFFA6JJ57MMRP62EQU6OFH2YODPFXAA, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:16.845346549 (604h22m16.716429584s) [upload=529718]: Next returned nil segment Dec 30 19:51:16.845364172 (604h22m16.716447207s) [upload=529718]: Waiting for error group managing segments... Dec 30 19:51:17.267522091 (604h22m17.138605126s) [noise=true , address=145.79.227.22:4532 , piece_id=34CFPGEIUKJAWIOHGCYT2IPARDF2Q6V4AZ75OEYJKWSGJP4XBLAA, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:17.381685254 (604h22m17.252768299s) [noise=true , address=145.79.227.14:4857 , piece_id=UAJPWWOJNMKWIR664WTJJTNLW6BSBWAXMBTM2KP27ZNDX5XUV7HA, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:17.491278576 (604h22m17.362361622s) [noise=true , address=77.90.31.16:4755 , piece_id=HSEFLVXWFD73XZ6FVA6L33OIZ4XMUD6KIC7OMUV2TZKSDJHBZIMQ, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:17.498649793 (604h22m17.369732838s) [noise=true , address=145.79.227.52:4462 , piece_id=ENQYW6ECH5WXOZROCR6AZGFFA6JJ57MMRP62EQU6OFH2YODPFXAA, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:17.508685632 (604h22m17.379768678s) [noise=true , address=77.90.31.12:4609 , piece_id=2SNAKTN72NMAYOAM5BNTNO6A6RD3XFIRNVVALVZLFRZIU47VEMXA, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:17.565292950 (604h22m17.436375986s) [noise=true , address=77.90.31.15:4710 , piece_id=HQYZ7QDCENX5OYIE4FJTF6VTC5RNQZPMH3KYEOBMYMB7EP2D3V2Q, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:17.587634352 (604h22m17.458717397s) [noise=true , address=45.91.116.8:4073 , piece_id=EV3LO3HZRKQF4A5R3HSPNCXKRE6BYLIQ2BQW6R5LMR4ROQQ6X3OA, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:17.655638912 (604h22m17.526721957s) [noise=true , address=45.91.116.7:4022 , piece_id=AMJKLUFDIY4M65Q4Y3DENPIQ2LP7IQDIBB5EJSXCMEE2MKPUOEDA, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:17.658922775 (604h22m17.53000581s) [noise=true , address=145.79.227.8:4359 , piece_id=XGBZZ64KTG2VJXGMOHZV5VY6STY2RJFFHANDSDXVY7UZ2DVWX3VA, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:17.678299712 (604h22m17.549382757s) [noise=true , address=45.91.116.8:4049 , piece_id=CDS23XTX2ILZKX7CWONNQ34F7W6G4BQHZOZLKFCWYJQKM3COPF4A, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:18.127069203 (604h22m17.998152249s) [noise=true , address=145.79.227.34:4937 , piece_id=GRDKPX53C5IPFCL2SE6L3NFCJRXUC7FPJ32HTFCCQGIIRUIITJ6Q, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:18.137212708 (604h22m18.008295743s) [noise=true , address=145.79.227.33:4904 , piece_id=KM563QF7HUNTIPTYGEPIM6DBCQSHZUYFLJQMGJ5DJDMWOWSKDB4A, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:18.160684672 (604h22m18.031767707s) [noise=true , address=145.79.227.40:4068 , piece_id=LY4JRNBNZBFHSKINNAGVFYVS2JXSOO6ELNKUPQMNPVY5LWZ7A4EQ, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:18.219837768 (604h22m18.090920833s) [noise=true , address=145.79.227.35:4487 , piece_id=YHVVDBISXCSLG3UW5XT7QEAKXOHSPS7RUY6HKZZS7NITA7OV47TQ, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:18.219866553 (604h22m18.090949598s) [seg_pos={0 0}, upload=529718]: Segment reached optimal threshold of 14 pieces. Dec 30 19:51:18.220084728 (604h22m18.091167763s) [noise=true , address=145.79.227.39:4040 , piece_id=OYYRO2KDG43DP2XYKFHWLSUJMSKQ6VC57GSW2PRNIA42XUTHEC2A, seg_pos={0 0}, upload=529718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gozJwRZvMK7Vbsdy775PtGynixvh3z5MHWwaLxzkrExseZosn): protocol: write tcp 188.166.225.170:44478->145.79.227.39:4040: use of closed network connection; write tcp 188.166.225.170:44478->145.79.227.39:4040: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44478->145.79.227.39:4040: use of closed network connection Dec 30 19:51:18.220092643 (604h22m18.091175678s) [noise=true , address=145.79.227.39:4040 , piece_id=OYYRO2KDG43DP2XYKFHWLSUJMSKQ6VC57GSW2PRNIA42XUTHEC2A, seg_pos={0 0}, upload=529718]: Long tail cancellation due to optimal threshold Dec 30 19:51:18.220170322 (604h22m18.091253357s) [noise=true , address=193.149.164.143:4704 , piece_id=LIPJT75FNGVN73FY277IE5FH6PXESYJXBCIUEVXJFQJ36QEHTH7A, seg_pos={0 0}, upload=529718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 30 19:51:18.220186161 (604h22m18.091269196s) [noise=true , address=193.149.164.143:4704 , piece_id=LIPJT75FNGVN73FY277IE5FH6PXESYJXBCIUEVXJFQJ36QEHTH7A, seg_pos={0 0}, upload=529718]: Long tail cancellation due to optimal threshold Dec 30 19:51:18.220218072 (604h22m18.091301107s) [noise=true , address=145.79.227.41:4099 , piece_id=IZN35ZJKWHB7DUAUFJ55HFUNJR5UMQ52YQN3CMYTTZUXMKJKN5UQ, seg_pos={0 0}, upload=529718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122dGFWNmsuheRmfEfNGWHftPpCcWv7HHXKymgJxsye6P7ZKhZT): protocol: write tcp 188.166.225.170:36572->145.79.227.41:4099: use of closed network connection; write tcp 188.166.225.170:36572->145.79.227.41:4099: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36572->145.79.227.41:4099: use of closed network connection Dec 30 19:51:18.220225947 (604h22m18.091308982s) [noise=true , address=145.79.227.41:4099 , piece_id=IZN35ZJKWHB7DUAUFJ55HFUNJR5UMQ52YQN3CMYTTZUXMKJKN5UQ, seg_pos={0 0}, upload=529718]: Long tail cancellation due to optimal threshold Dec 30 19:51:18.220496693 (604h22m18.091579728s) [noise=true , address=77.90.31.10:4526 , piece_id=EGGUMVBH572S2LZZWUYBKX2JSH2NVXC2BLHLXOKFUZKDMXHNPG4A, seg_pos={0 0}, upload=529718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: expected piece hash; context canceled; EOF Dec 30 19:51:18.220503786 (604h22m18.091586821s) [noise=true , address=77.90.31.10:4526 , piece_id=EGGUMVBH572S2LZZWUYBKX2JSH2NVXC2BLHLXOKFUZKDMXHNPG4A, seg_pos={0 0}, upload=529718]: Long tail cancellation due to optimal threshold Dec 30 19:51:18.220523634 (604h22m18.091606669s) [seg_pos={0 0}, upload=529718]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 19:51:18.220603907 (604h22m18.091686942s) [upload=529718]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:51:18.220611932 (604h22m18.091694967s) [upload=529718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:51:18.528318700 (604h22m18.399401735s) [upload=529718]: Done uploading segments... Dec 30 19:51:18.528354518 (604h22m18.399437553s) [upload=529718]: Upload finished. err: <nil> Duration: 24.192642456s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........X..X.....................X......................X] duration: 1m0.004074711s trace: https://jaeger.dev.storj.io/trace/fd3a56936ae71f2 FAIL (2025-12-30 19:52:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== piece_id=AMJKLUFDIY4M65Q4Y3DENPIQ2LP7IQDIBB5EJSXCMEE2MKPUOEDA, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.407338347 (604h22m15.278421392s) [noise=true , address=77.90.31.10:4526 , piece_id=EGGUMVBH572S2LZZWUYBKX2JSH2NVXC2BLHLXOKFUZKDMXHNPG4A, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.407626666 (604h22m15.278709711s) [noise=true , address=145.79.227.35:4487 , piece_id=YHVVDBISXCSLG3UW5XT7QEAKXOHSPS7RUY6HKZZS7NITA7OV47TQ, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.407809223 (604h22m15.278892258s) [noise=true , address=145.79.227.39:4040 , piece_id=OYYRO2KDG43DP2XYKFHWLSUJMSKQ6VC57GSW2PRNIA42XUTHEC2A, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.408000178 (604h22m15.279083213s) [noise=true , address=145.79.227.22:4532 , piece_id=34CFPGEIUKJAWIOHGCYT2IPARDF2Q6V4AZ75OEYJKWSGJP4XBLAA, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.408215778 (604h22m15.279298813s) [noise=true , address=77.90.31.16:4755 , piece_id=HSEFLVXWFD73XZ6FVA6L33OIZ4XMUD6KIC7OMUV2TZKSDJHBZIMQ, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.408338902 (604h22m15.279421937s) [noise=true , address=145.79.227.14:4857 , piece_id=UAJPWWOJNMKWIR664WTJJTNLW6BSBWAXMBTM2KP27ZNDX5XUV7HA, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.409170024 (604h22m15.28025306s) [noise=true , address=193.149.164.143:4704 , piece_id=LIPJT75FNGVN73FY277IE5FH6PXESYJXBCIUEVXJFQJ36QEHTH7A, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.410288113 (604h22m15.281371148s) [noise=true , address=145.79.227.34:4937 , piece_id=GRDKPX53C5IPFCL2SE6L3NFCJRXUC7FPJ32HTFCCQGIIRUIITJ6Q, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.411612275 (604h22m15.28269531s) [noise=true , address=45.91.116.8:4049 , piece_id=CDS23XTX2ILZKX7CWONNQ34F7W6G4BQHZOZLKFCWYJQKM3COPF4A, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.413868601 (604h22m15.284951646s) [noise=true , address=45.91.116.8:4073 , piece_id=EV3LO3HZRKQF4A5R3HSPNCXKRE6BYLIQ2BQW6R5LMR4ROQQ6X3OA, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.414897070 (604h22m15.285980115s) [noise=true , address=145.79.227.33:4904 , piece_id=KM563QF7HUNTIPTYGEPIM6DBCQSHZUYFLJQMGJ5DJDMWOWSKDB4A, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.415264499 (604h22m15.286347544s) [noise=true , address=77.90.31.12:4609 , piece_id=2SNAKTN72NMAYOAM5BNTNO6A6RD3XFIRNVVALVZLFRZIU47VEMXA, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.416022122 (604h22m15.287105167s) [noise=true , address=145.79.227.40:4068 , piece_id=LY4JRNBNZBFHSKINNAGVFYVS2JXSOO6ELNKUPQMNPVY5LWZ7A4EQ, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:15.418479571 (604h22m15.289562606s) [noise=true , address=145.79.227.52:4462 , piece_id=ENQYW6ECH5WXOZROCR6AZGFFA6JJ57MMRP62EQU6OFH2YODPFXAA, seg_pos={0 0}, upload=529718]: Uploading piece... Dec 30 19:51:16.845346549 (604h22m16.716429584s) [upload=529718]: Next returned nil segment Dec 30 19:51:16.845364172 (604h22m16.716447207s) [upload=529718]: Waiting for error group managing segments... Dec 30 19:51:17.267522091 (604h22m17.138605126s) [noise=true , address=145.79.227.22:4532 , piece_id=34CFPGEIUKJAWIOHGCYT2IPARDF2Q6V4AZ75OEYJKWSGJP4XBLAA, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:17.381685254 (604h22m17.252768299s) [noise=true , address=145.79.227.14:4857 , piece_id=UAJPWWOJNMKWIR664WTJJTNLW6BSBWAXMBTM2KP27ZNDX5XUV7HA, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:17.491278576 (604h22m17.362361622s) [noise=true , address=77.90.31.16:4755 , piece_id=HSEFLVXWFD73XZ6FVA6L33OIZ4XMUD6KIC7OMUV2TZKSDJHBZIMQ, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:17.498649793 (604h22m17.369732838s) [noise=true , address=145.79.227.52:4462 , piece_id=ENQYW6ECH5WXOZROCR6AZGFFA6JJ57MMRP62EQU6OFH2YODPFXAA, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:17.508685632 (604h22m17.379768678s) [noise=true , address=77.90.31.12:4609 , piece_id=2SNAKTN72NMAYOAM5BNTNO6A6RD3XFIRNVVALVZLFRZIU47VEMXA, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:17.565292950 (604h22m17.436375986s) [noise=true , address=77.90.31.15:4710 , piece_id=HQYZ7QDCENX5OYIE4FJTF6VTC5RNQZPMH3KYEOBMYMB7EP2D3V2Q, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:17.587634352 (604h22m17.458717397s) [noise=true , address=45.91.116.8:4073 , piece_id=EV3LO3HZRKQF4A5R3HSPNCXKRE6BYLIQ2BQW6R5LMR4ROQQ6X3OA, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:17.655638912 (604h22m17.526721957s) [noise=true , address=45.91.116.7:4022 , piece_id=AMJKLUFDIY4M65Q4Y3DENPIQ2LP7IQDIBB5EJSXCMEE2MKPUOEDA, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:17.658922775 (604h22m17.53000581s) [noise=true , address=145.79.227.8:4359 , piece_id=XGBZZ64KTG2VJXGMOHZV5VY6STY2RJFFHANDSDXVY7UZ2DVWX3VA, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:17.678299712 (604h22m17.549382757s) [noise=true , address=45.91.116.8:4049 , piece_id=CDS23XTX2ILZKX7CWONNQ34F7W6G4BQHZOZLKFCWYJQKM3COPF4A, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:18.127069203 (604h22m17.998152249s) [noise=true , address=145.79.227.34:4937 , piece_id=GRDKPX53C5IPFCL2SE6L3NFCJRXUC7FPJ32HTFCCQGIIRUIITJ6Q, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:18.137212708 (604h22m18.008295743s) [noise=true , address=145.79.227.33:4904 , piece_id=KM563QF7HUNTIPTYGEPIM6DBCQSHZUYFLJQMGJ5DJDMWOWSKDB4A, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:18.160684672 (604h22m18.031767707s) [noise=true , address=145.79.227.40:4068 , piece_id=LY4JRNBNZBFHSKINNAGVFYVS2JXSOO6ELNKUPQMNPVY5LWZ7A4EQ, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:18.219837768 (604h22m18.090920833s) [noise=true , address=145.79.227.35:4487 , piece_id=YHVVDBISXCSLG3UW5XT7QEAKXOHSPS7RUY6HKZZS7NITA7OV47TQ, seg_pos={0 0}, upload=529718]: Done uploading piece. err: <nil> Dec 30 19:51:18.219866553 (604h22m18.090949598s) [seg_pos={0 0}, upload=529718]: Segment reached optimal threshold of 14 pieces. Dec 30 19:51:18.220084728 (604h22m18.091167763s) [noise=true , address=145.79.227.39:4040 , piece_id=OYYRO2KDG43DP2XYKFHWLSUJMSKQ6VC57GSW2PRNIA42XUTHEC2A, seg_pos={0 0}, upload=529718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gozJwRZvMK7Vbsdy775PtGynixvh3z5MHWwaLxzkrExseZosn): protocol: write tcp 188.166.225.170:44478->145.79.227.39:4040: use of closed network connection; write tcp 188.166.225.170:44478->145.79.227.39:4040: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44478->145.79.227.39:4040: use of closed network connection Dec 30 19:51:18.220092643 (604h22m18.091175678s) [noise=true , address=145.79.227.39:4040 , piece_id=OYYRO2KDG43DP2XYKFHWLSUJMSKQ6VC57GSW2PRNIA42XUTHEC2A, seg_pos={0 0}, upload=529718]: Long tail cancellation due to optimal threshold Dec 30 19:51:18.220170322 (604h22m18.091253357s) [noise=true , address=193.149.164.143:4704 , piece_id=LIPJT75FNGVN73FY277IE5FH6PXESYJXBCIUEVXJFQJ36QEHTH7A, seg_pos={0 0}, upload=529718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUVmWsfAfQiPWhDkYRNGhims8TLQqGqvhrZMYugLLTdmLk2xY): protocol: expected piece hash; context canceled; EOF Dec 30 19:51:18.220186161 (604h22m18.091269196s) [noise=true , address=193.149.164.143:4704 , piece_id=LIPJT75FNGVN73FY277IE5FH6PXESYJXBCIUEVXJFQJ36QEHTH7A, seg_pos={0 0}, upload=529718]: Long tail cancellation due to optimal threshold Dec 30 19:51:18.220218072 (604h22m18.091301107s) [noise=true , address=145.79.227.41:4099 , piece_id=IZN35ZJKWHB7DUAUFJ55HFUNJR5UMQ52YQN3CMYTTZUXMKJKN5UQ, seg_pos={0 0}, upload=529718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:122dGFWNmsuheRmfEfNGWHftPpCcWv7HHXKymgJxsye6P7ZKhZT): protocol: write tcp 188.166.225.170:36572->145.79.227.41:4099: use of closed network connection; write tcp 188.166.225.170:36572->145.79.227.41:4099: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36572->145.79.227.41:4099: use of closed network connection Dec 30 19:51:18.220225947 (604h22m18.091308982s) [noise=true , address=145.79.227.41:4099 , piece_id=IZN35ZJKWHB7DUAUFJ55HFUNJR5UMQ52YQN3CMYTTZUXMKJKN5UQ, seg_pos={0 0}, upload=529718]: Long tail cancellation due to optimal threshold Dec 30 19:51:18.220496693 (604h22m18.091579728s) [noise=true , address=77.90.31.10:4526 , piece_id=EGGUMVBH572S2LZZWUYBKX2JSH2NVXC2BLHLXOKFUZKDMXHNPG4A, seg_pos={0 0}, upload=529718]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: expected piece hash; context canceled; EOF Dec 30 19:51:18.220503786 (604h22m18.091586821s) [noise=true , address=77.90.31.10:4526 , piece_id=EGGUMVBH572S2LZZWUYBKX2JSH2NVXC2BLHLXOKFUZKDMXHNPG4A, seg_pos={0 0}, upload=529718]: Long tail cancellation due to optimal threshold Dec 30 19:51:18.220523634 (604h22m18.091606669s) [seg_pos={0 0}, upload=529718]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 19:51:18.220603907 (604h22m18.091686942s) [upload=529718]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:51:18.220611932 (604h22m18.091694967s) [upload=529718]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:51:18.528318700 (604h22m18.399401735s) [upload=529718]: Done uploading segments... Dec 30 19:51:18.528354518 (604h22m18.399437553s) [upload=529718]: Upload finished. err: <nil> Duration: 1m0.004097734s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..X.....X..XX...X..X.....XX.X.....X.....X.................X.] duration: 6.599228637s trace: https://jaeger.dev.storj.io/trace/77d763626a2de69f FAIL (2025-12-30 19:51:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== . Dec 30 19:50:49.706686384 (604h21m49.577769419s) [upload=529713]: Got next segment. Inline: false Dec 30 19:50:49.706728235 (604h21m49.57781127s) [upload=529713]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 19:50:49.706732713 (604h21m49.577815748s) [upload=529713]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 19:50:49.956439120 (604h21m49.827522155s) [seg_pos={0 0}, upload=529713]: Begin upload segment... Dec 30 19:50:49.956496509 (604h21m49.827579554s) [seg_pos={0 0}, upload=529713]: Done begin upload segment. Dec 30 19:50:49.956533911 (604h21m49.827616956s) [noise=true , address=145.79.227.17:4970 , piece_id=4JLGJNJ5NY6EKS263RE7Q4MZORF5TLLD25GSPVIOBYZDP2XEHVBA, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.956666412 (604h21m49.827749447s) [noise=true , address=145.79.227.51:4411 , piece_id=4FPWDJJUOXZJKPIADGCCDACHFPIECD6JTL6B3MRNX2767TD6AYXA, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.957306692 (604h21m49.828389727s) [noise=true , address=77.90.31.10:4517 , piece_id=N7PNEZJHTQIRH2H5JKUCTGYEUQNRUVAG7E3ZXSPLOSY4YB7NOPRA, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.957724868 (604h21m49.828807913s) [noise=true , address=145.79.227.39:4014 , piece_id=MFTOYKF3Q7TJO6GARDKQ4CZTJCDORXXB5WCNU67A27KMNLBFDCIA, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.957888850 (604h21m49.828971895s) [noise=true , address=145.79.227.11:4732 , piece_id=7A6NUULRBP4EKSCZ6K72XREWNXSCLEMKIEP5DMMQZBJBDFICEXUA, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.958315583 (604h21m49.829398628s) [noise=true , address=145.79.227.23:4546 , piece_id=EABR4DPGHN5R5JSGDYHOTFU5TUXDC3UVUVMJQI4TCXXFTMYYRT5Q, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.958450761 (604h21m49.829533796s) [noise=true , address=77.90.31.21:4878 , piece_id=AUE3ABLPLC4B6NR7TQ4X2RUB4LOWCQPMYVGBQYEZ2HFQUIN2ZJWA, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.958822308 (604h21m49.829905343s) [noise=true , address=77.90.31.16:4735 , piece_id=MO6I2KLJBXFRZ5BOUHUQF3ABPPMTRNPMOUYLXLWDG2XBEYSU6YGA, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.958932247 (604h21m49.830015292s) [noise=true , address=145.79.227.2:4113 , piece_id=7GKO34YXFFS3HXEG2YTAW6WXGNHQG5J7FD6IMEJIRIUHHLUBK7WQ, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.959417631 (604h21m49.830500676s) [noise=true , address=145.79.227.31:4799 , piece_id=LDIWHZNNXSWR4Y6EPQVNDFKABV2427WXOVDDOXAP3ABURMTZPK3A, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.959438411 (604h21m49.830521446s) [noise=true , address=45.91.116.10:4084 , piece_id=RUJXLLDM5FCYCGQEYOF2MI4EAYOD5LVQDRGZB2L2RMK5YQ5B6ZKQ, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.959901042 (604h21m49.830984087s) [noise=true , address=145.79.227.41:4085 , piece_id=RPIMMFDMQM24OFDLCQAWC2GWS5WKFFUFPNIAMIZKWZZS4RF54W7Q, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.960008917 (604h21m49.831091962s) [noise=true , address=45.91.116.13:4154 , piece_id=UHJB6FDP4BX7JEHZXKDDFMYVPHKTEUQI4WAHKMHIKSKEH6VYI42A, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.960389352 (604h21m49.831472397s) [noise=true , address=45.91.116.12:4226 , piece_id=HMHO3QMYPUCSBXCN2JILSDY5DVXCYXS25SYHSHB3EGHWVD4KNOUA, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.960565286 (604h21m49.831648332s) [noise=true , address=145.79.227.15:4873 , piece_id=7NH5E4SMR5JPZLAS7Q7PUCBHFE3AAOWGPGYTWVBT5IQBZBBTLTQQ, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.960897439 (604h21m49.831980474s) [noise=true , address=145.79.227.33:4870 , piece_id=RSW5PU23ZUPQGLTRM62SVGGLGVTIPLBSR5ZARGXJ3DVOC2UPM4EA, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:51.646909056 (604h21m51.517992101s) [upload=529713]: Next returned nil segment Dec 30 19:50:51.646928483 (604h21m51.518011529s) [upload=529713]: Waiting for error group managing segments... Dec 30 19:50:51.934292786 (604h21m51.805375821s) [noise=true , address=145.79.227.31:4799 , piece_id=LDIWHZNNXSWR4Y6EPQVNDFKABV2427WXOVDDOXAP3ABURMTZPK3A, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:51.940462203 (604h21m51.811545238s) [noise=true , address=145.79.227.51:4411 , piece_id=4FPWDJJUOXZJKPIADGCCDACHFPIECD6JTL6B3MRNX2767TD6AYXA, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:51.975314537 (604h21m51.846397572s) [noise=true , address=145.79.227.11:4732 , piece_id=7A6NUULRBP4EKSCZ6K72XREWNXSCLEMKIEP5DMMQZBJBDFICEXUA, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:51.986605326 (604h21m51.857688361s) [noise=true , address=145.79.227.17:4970 , piece_id=4JLGJNJ5NY6EKS263RE7Q4MZORF5TLLD25GSPVIOBYZDP2XEHVBA, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:52.026019757 (604h21m51.897102802s) [noise=true , address=145.79.227.41:4085 , piece_id=RPIMMFDMQM24OFDLCQAWC2GWS5WKFFUFPNIAMIZKWZZS4RF54W7Q, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:52.030867858 (604h21m51.901950893s) [noise=true , address=145.79.227.39:4014 , piece_id=MFTOYKF3Q7TJO6GARDKQ4CZTJCDORXXB5WCNU67A27KMNLBFDCIA, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:52.177525915 (604h21m52.04860896s) [noise=true , address=77.90.31.10:4517 , piece_id=N7PNEZJHTQIRH2H5JKUCTGYEUQNRUVAG7E3ZXSPLOSY4YB7NOPRA, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:52.202966258 (604h21m52.074049293s) [noise=true , address=77.90.31.16:4735 , piece_id=MO6I2KLJBXFRZ5BOUHUQF3ABPPMTRNPMOUYLXLWDG2XBEYSU6YGA, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:52.219992708 (604h21m52.091075743s) [noise=true , address=77.90.31.21:4878 , piece_id=AUE3ABLPLC4B6NR7TQ4X2RUB4LOWCQPMYVGBQYEZ2HFQUIN2ZJWA, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:52.230964761 (604h21m52.102047796s) [noise=true , address=145.79.227.15:4873 , piece_id=7NH5E4SMR5JPZLAS7Q7PUCBHFE3AAOWGPGYTWVBT5IQBZBBTLTQQ, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:52.274453189 (604h21m52.145536234s) [noise=true , address=45.91.116.12:4226 , piece_id=HMHO3QMYPUCSBXCN2JILSDY5DVXCYXS25SYHSHB3EGHWVD4KNOUA, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:52.302329220 (604h21m52.173412255s) [noise=true , address=45.91.116.13:4154 , piece_id=UHJB6FDP4BX7JEHZXKDDFMYVPHKTEUQI4WAHKMHIKSKEH6VYI42A, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:52.364019437 (604h21m52.235102482s) [noise=true , address=45.91.116.10:4084 , piece_id=RUJXLLDM5FCYCGQEYOF2MI4EAYOD5LVQDRGZB2L2RMK5YQ5B6ZKQ, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:52.364038934 (604h21m52.235121969s) [seg_pos={0 0}, upload=529713]: Segment reached optimal threshold of 13 pieces. Dec 30 19:50:52.364227393 (604h21m52.235310438s) [noise=true , address=145.79.227.23:4546 , piece_id=EABR4DPGHN5R5JSGDYHOTFU5TUXDC3UVUVMJQI4TCXXFTMYYRT5Q, seg_pos={0 0}, upload=529713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: write tcp 188.166.225.170:51884->145.79.227.23:4546: use of closed network connection; write tcp 188.166.225.170:51884->145.79.227.23:4546: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51884->145.79.227.23:4546: use of closed network connection Dec 30 19:50:52.364235579 (604h21m52.235318614s) [noise=true , address=145.79.227.23:4546 , piece_id=EABR4DPGHN5R5JSGDYHOTFU5TUXDC3UVUVMJQI4TCXXFTMYYRT5Q, seg_pos={0 0}, upload=529713]: Long tail cancellation due to optimal threshold Dec 30 19:50:52.364465657 (604h21m52.235548692s) [noise=true , address=145.79.227.2:4113 , piece_id=7GKO34YXFFS3HXEG2YTAW6WXGNHQG5J7FD6IMEJIRIUHHLUBK7WQ, seg_pos={0 0}, upload=529713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:38802->145.79.227.2:4113: use of closed network connection; write tcp 188.166.225.170:38802->145.79.227.2:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38802->145.79.227.2:4113: use of closed network connection Dec 30 19:50:52.364474493 (604h21m52.235557538s) [noise=true , address=145.79.227.2:4113 , piece_id=7GKO34YXFFS3HXEG2YTAW6WXGNHQG5J7FD6IMEJIRIUHHLUBK7WQ, seg_pos={0 0}, upload=529713]: Long tail cancellation due to optimal threshold Dec 30 19:50:52.364541711 (604h21m52.235624746s) [noise=true , address=145.79.227.33:4870 , piece_id=RSW5PU23ZUPQGLTRM62SVGGLGVTIPLBSR5ZARGXJ3DVOC2UPM4EA, seg_pos={0 0}, upload=529713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 19:50:52.364561259 (604h21m52.235644304s) [noise=true , address=145.79.227.33:4870 , piece_id=RSW5PU23ZUPQGLTRM62SVGGLGVTIPLBSR5ZARGXJ3DVOC2UPM4EA, seg_pos={0 0}, upload=529713]: Long tail cancellation due to optimal threshold Dec 30 19:50:52.364605673 (604h21m52.235688708s) [seg_pos={0 0}, upload=529713]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 19:50:52.364730571 (604h21m52.235813617s) [upload=529713]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:50:52.364736002 (604h21m52.235819047s) [upload=529713]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:50:52.655672157 (604h21m52.526755202s) [upload=529713]: Done uploading segments... Dec 30 19:50:52.655698627 (604h21m52.526781682s) [upload=529713]: Upload finished. err: <nil> Duration: 6.599258684s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X..X.....X..XX...X..X.....XX.X.....X.....X.................X] duration: 30.002556912s trace: https://jaeger.dev.storj.io/trace/77d763626a2de69f FAIL (2025-12-30 19:51:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== . Dec 30 19:50:49.706686384 (604h21m49.577769419s) [upload=529713]: Got next segment. Inline: false Dec 30 19:50:49.706728235 (604h21m49.57781127s) [upload=529713]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 19:50:49.706732713 (604h21m49.577815748s) [upload=529713]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 19:50:49.956439120 (604h21m49.827522155s) [seg_pos={0 0}, upload=529713]: Begin upload segment... Dec 30 19:50:49.956496509 (604h21m49.827579554s) [seg_pos={0 0}, upload=529713]: Done begin upload segment. Dec 30 19:50:49.956533911 (604h21m49.827616956s) [noise=true , address=145.79.227.17:4970 , piece_id=4JLGJNJ5NY6EKS263RE7Q4MZORF5TLLD25GSPVIOBYZDP2XEHVBA, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.956666412 (604h21m49.827749447s) [noise=true , address=145.79.227.51:4411 , piece_id=4FPWDJJUOXZJKPIADGCCDACHFPIECD6JTL6B3MRNX2767TD6AYXA, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.957306692 (604h21m49.828389727s) [noise=true , address=77.90.31.10:4517 , piece_id=N7PNEZJHTQIRH2H5JKUCTGYEUQNRUVAG7E3ZXSPLOSY4YB7NOPRA, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.957724868 (604h21m49.828807913s) [noise=true , address=145.79.227.39:4014 , piece_id=MFTOYKF3Q7TJO6GARDKQ4CZTJCDORXXB5WCNU67A27KMNLBFDCIA, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.957888850 (604h21m49.828971895s) [noise=true , address=145.79.227.11:4732 , piece_id=7A6NUULRBP4EKSCZ6K72XREWNXSCLEMKIEP5DMMQZBJBDFICEXUA, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.958315583 (604h21m49.829398628s) [noise=true , address=145.79.227.23:4546 , piece_id=EABR4DPGHN5R5JSGDYHOTFU5TUXDC3UVUVMJQI4TCXXFTMYYRT5Q, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.958450761 (604h21m49.829533796s) [noise=true , address=77.90.31.21:4878 , piece_id=AUE3ABLPLC4B6NR7TQ4X2RUB4LOWCQPMYVGBQYEZ2HFQUIN2ZJWA, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.958822308 (604h21m49.829905343s) [noise=true , address=77.90.31.16:4735 , piece_id=MO6I2KLJBXFRZ5BOUHUQF3ABPPMTRNPMOUYLXLWDG2XBEYSU6YGA, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.958932247 (604h21m49.830015292s) [noise=true , address=145.79.227.2:4113 , piece_id=7GKO34YXFFS3HXEG2YTAW6WXGNHQG5J7FD6IMEJIRIUHHLUBK7WQ, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.959417631 (604h21m49.830500676s) [noise=true , address=145.79.227.31:4799 , piece_id=LDIWHZNNXSWR4Y6EPQVNDFKABV2427WXOVDDOXAP3ABURMTZPK3A, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.959438411 (604h21m49.830521446s) [noise=true , address=45.91.116.10:4084 , piece_id=RUJXLLDM5FCYCGQEYOF2MI4EAYOD5LVQDRGZB2L2RMK5YQ5B6ZKQ, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.959901042 (604h21m49.830984087s) [noise=true , address=145.79.227.41:4085 , piece_id=RPIMMFDMQM24OFDLCQAWC2GWS5WKFFUFPNIAMIZKWZZS4RF54W7Q, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.960008917 (604h21m49.831091962s) [noise=true , address=45.91.116.13:4154 , piece_id=UHJB6FDP4BX7JEHZXKDDFMYVPHKTEUQI4WAHKMHIKSKEH6VYI42A, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.960389352 (604h21m49.831472397s) [noise=true , address=45.91.116.12:4226 , piece_id=HMHO3QMYPUCSBXCN2JILSDY5DVXCYXS25SYHSHB3EGHWVD4KNOUA, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.960565286 (604h21m49.831648332s) [noise=true , address=145.79.227.15:4873 , piece_id=7NH5E4SMR5JPZLAS7Q7PUCBHFE3AAOWGPGYTWVBT5IQBZBBTLTQQ, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:49.960897439 (604h21m49.831980474s) [noise=true , address=145.79.227.33:4870 , piece_id=RSW5PU23ZUPQGLTRM62SVGGLGVTIPLBSR5ZARGXJ3DVOC2UPM4EA, seg_pos={0 0}, upload=529713]: Uploading piece... Dec 30 19:50:51.646909056 (604h21m51.517992101s) [upload=529713]: Next returned nil segment Dec 30 19:50:51.646928483 (604h21m51.518011529s) [upload=529713]: Waiting for error group managing segments... Dec 30 19:50:51.934292786 (604h21m51.805375821s) [noise=true , address=145.79.227.31:4799 , piece_id=LDIWHZNNXSWR4Y6EPQVNDFKABV2427WXOVDDOXAP3ABURMTZPK3A, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:51.940462203 (604h21m51.811545238s) [noise=true , address=145.79.227.51:4411 , piece_id=4FPWDJJUOXZJKPIADGCCDACHFPIECD6JTL6B3MRNX2767TD6AYXA, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:51.975314537 (604h21m51.846397572s) [noise=true , address=145.79.227.11:4732 , piece_id=7A6NUULRBP4EKSCZ6K72XREWNXSCLEMKIEP5DMMQZBJBDFICEXUA, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:51.986605326 (604h21m51.857688361s) [noise=true , address=145.79.227.17:4970 , piece_id=4JLGJNJ5NY6EKS263RE7Q4MZORF5TLLD25GSPVIOBYZDP2XEHVBA, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:52.026019757 (604h21m51.897102802s) [noise=true , address=145.79.227.41:4085 , piece_id=RPIMMFDMQM24OFDLCQAWC2GWS5WKFFUFPNIAMIZKWZZS4RF54W7Q, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:52.030867858 (604h21m51.901950893s) [noise=true , address=145.79.227.39:4014 , piece_id=MFTOYKF3Q7TJO6GARDKQ4CZTJCDORXXB5WCNU67A27KMNLBFDCIA, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:52.177525915 (604h21m52.04860896s) [noise=true , address=77.90.31.10:4517 , piece_id=N7PNEZJHTQIRH2H5JKUCTGYEUQNRUVAG7E3ZXSPLOSY4YB7NOPRA, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:52.202966258 (604h21m52.074049293s) [noise=true , address=77.90.31.16:4735 , piece_id=MO6I2KLJBXFRZ5BOUHUQF3ABPPMTRNPMOUYLXLWDG2XBEYSU6YGA, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:52.219992708 (604h21m52.091075743s) [noise=true , address=77.90.31.21:4878 , piece_id=AUE3ABLPLC4B6NR7TQ4X2RUB4LOWCQPMYVGBQYEZ2HFQUIN2ZJWA, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:52.230964761 (604h21m52.102047796s) [noise=true , address=145.79.227.15:4873 , piece_id=7NH5E4SMR5JPZLAS7Q7PUCBHFE3AAOWGPGYTWVBT5IQBZBBTLTQQ, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:52.274453189 (604h21m52.145536234s) [noise=true , address=45.91.116.12:4226 , piece_id=HMHO3QMYPUCSBXCN2JILSDY5DVXCYXS25SYHSHB3EGHWVD4KNOUA, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:52.302329220 (604h21m52.173412255s) [noise=true , address=45.91.116.13:4154 , piece_id=UHJB6FDP4BX7JEHZXKDDFMYVPHKTEUQI4WAHKMHIKSKEH6VYI42A, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:52.364019437 (604h21m52.235102482s) [noise=true , address=45.91.116.10:4084 , piece_id=RUJXLLDM5FCYCGQEYOF2MI4EAYOD5LVQDRGZB2L2RMK5YQ5B6ZKQ, seg_pos={0 0}, upload=529713]: Done uploading piece. err: <nil> Dec 30 19:50:52.364038934 (604h21m52.235121969s) [seg_pos={0 0}, upload=529713]: Segment reached optimal threshold of 13 pieces. Dec 30 19:50:52.364227393 (604h21m52.235310438s) [noise=true , address=145.79.227.23:4546 , piece_id=EABR4DPGHN5R5JSGDYHOTFU5TUXDC3UVUVMJQI4TCXXFTMYYRT5Q, seg_pos={0 0}, upload=529713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZMTFpqfMjeVWXJuvsGdbGkMtGQn6axUZFoU1N9sK6fnbPcMy5): protocol: write tcp 188.166.225.170:51884->145.79.227.23:4546: use of closed network connection; write tcp 188.166.225.170:51884->145.79.227.23:4546: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51884->145.79.227.23:4546: use of closed network connection Dec 30 19:50:52.364235579 (604h21m52.235318614s) [noise=true , address=145.79.227.23:4546 , piece_id=EABR4DPGHN5R5JSGDYHOTFU5TUXDC3UVUVMJQI4TCXXFTMYYRT5Q, seg_pos={0 0}, upload=529713]: Long tail cancellation due to optimal threshold Dec 30 19:50:52.364465657 (604h21m52.235548692s) [noise=true , address=145.79.227.2:4113 , piece_id=7GKO34YXFFS3HXEG2YTAW6WXGNHQG5J7FD6IMEJIRIUHHLUBK7WQ, seg_pos={0 0}, upload=529713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV): protocol: write tcp 188.166.225.170:38802->145.79.227.2:4113: use of closed network connection; write tcp 188.166.225.170:38802->145.79.227.2:4113: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38802->145.79.227.2:4113: use of closed network connection Dec 30 19:50:52.364474493 (604h21m52.235557538s) [noise=true , address=145.79.227.2:4113 , piece_id=7GKO34YXFFS3HXEG2YTAW6WXGNHQG5J7FD6IMEJIRIUHHLUBK7WQ, seg_pos={0 0}, upload=529713]: Long tail cancellation due to optimal threshold Dec 30 19:50:52.364541711 (604h21m52.235624746s) [noise=true , address=145.79.227.33:4870 , piece_id=RSW5PU23ZUPQGLTRM62SVGGLGVTIPLBSR5ZARGXJ3DVOC2UPM4EA, seg_pos={0 0}, upload=529713]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 19:50:52.364561259 (604h21m52.235644304s) [noise=true , address=145.79.227.33:4870 , piece_id=RSW5PU23ZUPQGLTRM62SVGGLGVTIPLBSR5ZARGXJ3DVOC2UPM4EA, seg_pos={0 0}, upload=529713]: Long tail cancellation due to optimal threshold Dec 30 19:50:52.364605673 (604h21m52.235688708s) [seg_pos={0 0}, upload=529713]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 19:50:52.364730571 (604h21m52.235813617s) [upload=529713]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:50:52.364736002 (604h21m52.235819047s) [upload=529713]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:50:52.655672157 (604h21m52.526755202s) [upload=529713]: Done uploading segments... Dec 30 19:50:52.655698627 (604h21m52.526781682s) [upload=529713]: Upload finished. err: <nil> Duration: 30.002583543s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X..X.....................X......................] duration: 29.396824061s trace: https://jaeger.dev.storj.io/trace/2e4bd3c87074131f FAIL (2025-12-30 19:51:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== 98432889s) [noise=true , address=145.79.227.49:4362 , piece_id=UXLOT5FU522U73SMQEUIZ4K43WTQEAV5XRVSGG6EWQON43UWQAYQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.827625579 (604h4m49.698708614s) [noise=true , address=145.79.227.4:4218 , piece_id=2H4DI7772BR4JLLGKJYTSKTKVOABDB4T6UJ74OBTLXHV23WPN5HA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.827942963 (604h4m49.699025998s) [noise=true , address=45.91.116.9:4288 , piece_id=QLRNLFOW2RZ6JLJDNJEKBQ6JWJLOZ57NBPADF3KYVLL4L5FTC62A, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.828225852 (604h4m49.699308887s) [noise=true , address=45.91.116.14:4199 , piece_id=J3LDA3N2Q6B3VTWS5LG25LUD5E2L7I3FD56JPOVJVCJYLXC3AKXQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.828502729 (604h4m49.699585764s) [noise=true , address=64.49.13.138:4483 , piece_id=GRAGOWDG4I5TDHVNX75QJBTS7NIIRY3B73VWDO7DWNITNJXBAYWQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.828558776 (604h4m49.699641821s) [noise=true , address=64.49.13.169:4708 , piece_id=BXCIK6V4R5LZHWUTJOCL66442ZM6T7IYWD2OH5OM2SM7GX5JBIZA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.828853557 (604h4m49.699936592s) [noise=true , address=145.79.227.50:4384 , piece_id=U5R4PT66AXFJFPAGFADCVX6NNNIFLFM3CFTREKVEOOUDHTVTUTBA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.829160732 (604h4m49.700243767s) [noise=true , address=5.231.73.4:4027 , piece_id=DVNFM76WSUGT65ZKJY4RFVXXLZIDQJ7TQ5CYBPUMZ2GXQWJOIY3Q, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.829534374 (604h4m49.700617409s) [noise=true , address=145.79.227.52:4458 , piece_id=2XCSF7MRBBZBGEJ2GE63K25BO5DFB7DUSZ4KXMFRS5NOOHPSEJRA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.829873299 (604h4m49.700956334s) [noise=true , address=145.79.227.34:4908 , piece_id=X57UEZNRHXKLR7R3VEWRKCALEHCMTE7NR73QGALOEZ7NKOBKDDFA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.830200823 (604h4m49.701283868s) [noise=true , address=145.79.227.33:4887 , piece_id=I7CHA6ZOBOS7J4SHID6MQLGAVR66BZ2D2J4VSTFCQW525BPN5KHQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.838625865 (604h4m49.7097089s) [noise=true , address=145.79.227.3:4180 , piece_id=CYOSWRJR37YSHNJBWZZPMOLM7TVRJ2GFEV4XNNHRAQJWFU6Y2JIQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.839781686 (604h4m49.710864731s) [noise=true , address=145.79.227.36:4546 , piece_id=4V7GWX3MULB6CR3QH44ZIOISDONXKJ334XFSYHXUECJZBMRHQCKA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.840846864 (604h4m49.711929899s) [noise=true , address=77.90.31.14:4669 , piece_id=O2Z45BXHUPJQZZ7EJG4FKS3MNDPHWPBT4EMZCZV2HTJSEVAI7WHQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.841958571 (604h4m49.713041606s) [noise=true , address=77.90.31.10:4534 , piece_id=KPMW47ZBIINLCZ67FH6EGEDFLEAT7F35CYP62D4TT7IBLQ57I2FA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:51.267701872 (604h4m51.138784917s) [upload=529467]: Next returned nil segment Dec 30 19:33:51.267719867 (604h4m51.138802912s) [upload=529467]: Waiting for error group managing segments... Dec 30 19:33:51.700925266 (604h4m51.572008302s) [noise=true , address=145.79.227.3:4180 , piece_id=CYOSWRJR37YSHNJBWZZPMOLM7TVRJ2GFEV4XNNHRAQJWFU6Y2JIQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.707415294 (604h4m51.578498329s) [noise=true , address=145.79.227.50:4384 , piece_id=U5R4PT66AXFJFPAGFADCVX6NNNIFLFM3CFTREKVEOOUDHTVTUTBA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.732565163 (604h4m51.603648198s) [noise=true , address=145.79.227.36:4546 , piece_id=4V7GWX3MULB6CR3QH44ZIOISDONXKJ334XFSYHXUECJZBMRHQCKA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.745717487 (604h4m51.616800522s) [noise=true , address=64.49.13.170:4717 , piece_id=NSYCFF26VLJ6QPF5JAPBZRBLULLDAWUEPHI5HZTPSXQVX43KQHVQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.782268034 (604h4m51.653351069s) [noise=true , address=64.49.13.138:4483 , piece_id=GRAGOWDG4I5TDHVNX75QJBTS7NIIRY3B73VWDO7DWNITNJXBAYWQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.856613327 (604h4m51.727696362s) [noise=true , address=145.79.227.52:4458 , piece_id=2XCSF7MRBBZBGEJ2GE63K25BO5DFB7DUSZ4KXMFRS5NOOHPSEJRA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.900758014 (604h4m51.771841059s) [noise=true , address=77.90.31.10:4534 , piece_id=KPMW47ZBIINLCZ67FH6EGEDFLEAT7F35CYP62D4TT7IBLQ57I2FA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.968806577 (604h4m51.839889622s) [noise=true , address=77.90.31.13:4631 , piece_id=IDYOBFP732AUWTYULX2I5ETBTS3DFAVW2EVEZ4BB5NYZFA4P2FTA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.000820185 (604h4m51.87190323s) [noise=true , address=45.91.116.14:4199 , piece_id=J3LDA3N2Q6B3VTWS5LG25LUD5E2L7I3FD56JPOVJVCJYLXC3AKXQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.111919172 (604h4m51.983002207s) [noise=true , address=145.79.227.34:4908 , piece_id=X57UEZNRHXKLR7R3VEWRKCALEHCMTE7NR73QGALOEZ7NKOBKDDFA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.513345511 (604h4m52.384428556s) [noise=true , address=5.231.73.4:4027 , piece_id=DVNFM76WSUGT65ZKJY4RFVXXLZIDQJ7TQ5CYBPUMZ2GXQWJOIY3Q, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.525239208 (604h4m52.396322253s) [noise=true , address=145.79.227.49:4362 , piece_id=UXLOT5FU522U73SMQEUIZ4K43WTQEAV5XRVSGG6EWQON43UWQAYQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.547688835 (604h4m52.41877187s) [noise=true , address=145.79.227.30:4793 , piece_id=37HWPUMM2Q4CZDYKXZIKL5KKPEH4O6ASHI675HEP2K7EOCGBVLFA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.777085496 (604h4m52.648168531s) [noise=true , address=145.79.227.33:4887 , piece_id=I7CHA6ZOBOS7J4SHID6MQLGAVR66BZ2D2J4VSTFCQW525BPN5KHQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.777110584 (604h4m52.648193619s) [seg_pos={0 0}, upload=529467]: Segment reached optimal threshold of 14 pieces. Dec 30 19:33:52.777320053 (604h4m52.648403088s) [noise=true , address=45.91.116.9:4288 , piece_id=QLRNLFOW2RZ6JLJDNJEKBQ6JWJLOZ57NBPADF3KYVLL4L5FTC62A, seg_pos={0 0}, upload=529467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX): protocol: write tcp 188.166.225.170:38490->45.91.116.9:4288: use of closed network connection; write tcp 188.166.225.170:38490->45.91.116.9:4288: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38490->45.91.116.9:4288: use of closed network connection Dec 30 19:33:52.777326334 (604h4m52.64840938s) [noise=true , address=45.91.116.9:4288 , piece_id=QLRNLFOW2RZ6JLJDNJEKBQ6JWJLOZ57NBPADF3KYVLL4L5FTC62A, seg_pos={0 0}, upload=529467]: Long tail cancellation due to optimal threshold Dec 30 19:33:52.777615194 (604h4m52.648698229s) [noise=true , address=77.90.31.14:4669 , piece_id=O2Z45BXHUPJQZZ7EJG4FKS3MNDPHWPBT4EMZCZV2HTJSEVAI7WHQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T): protocol: expected piece hash; context canceled; EOF Dec 30 19:33:52.777622820 (604h4m52.648705855s) [noise=true , address=77.90.31.14:4669 , piece_id=O2Z45BXHUPJQZZ7EJG4FKS3MNDPHWPBT4EMZCZV2HTJSEVAI7WHQ, seg_pos={0 0}, upload=529467]: Long tail cancellation due to optimal threshold Dec 30 19:33:52.777648788 (604h4m52.648731823s) [noise=true , address=64.49.13.169:4708 , piece_id=BXCIK6V4R5LZHWUTJOCL66442ZM6T7IYWD2OH5OM2SM7GX5JBIZA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:36148->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:36148->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36148->64.49.13.169:4708: use of closed network connection Dec 30 19:33:52.777665260 (604h4m52.648748295s) [noise=true , address=64.49.13.169:4708 , piece_id=BXCIK6V4R5LZHWUTJOCL66442ZM6T7IYWD2OH5OM2SM7GX5JBIZA, seg_pos={0 0}, upload=529467]: Long tail cancellation due to optimal threshold Dec 30 19:33:52.777872405 (604h4m52.64895545s) [noise=true , address=145.79.227.4:4218 , piece_id=2H4DI7772BR4JLLGKJYTSKTKVOABDB4T6UJ74OBTLXHV23WPN5HA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mbzxg1BjmbXeHxGgheZmQoN91k8NgaAUUASPkJe3fEmS4eEkc): protocol: expected piece hash; context canceled; EOF Dec 30 19:33:52.777879939 (604h4m52.648962984s) [noise=true , address=145.79.227.4:4218 , piece_id=2H4DI7772BR4JLLGKJYTSKTKVOABDB4T6UJ74OBTLXHV23WPN5HA, seg_pos={0 0}, upload=529467]: Long tail cancellation due to optimal threshold Dec 30 19:33:52.777898815 (604h4m52.64898185s) [seg_pos={0 0}, upload=529467]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 19:33:52.777974709 (604h4m52.649057755s) [upload=529467]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:33:52.777979669 (604h4m52.649062714s) [upload=529467]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:33:53.082585408 (604h4m52.953668453s) [upload=529467]: Done uploading segments... Dec 30 19:33:53.082605185 (604h4m52.953688221s) [upload=529467]: Upload finished. err: <nil> Duration: 29.396855572s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X..X.....................X.....................] duration: 26.676259188s trace: https://jaeger.dev.storj.io/trace/2e4bd3c87074131f FAIL (2025-12-30 19:50:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== 98432889s) [noise=true , address=145.79.227.49:4362 , piece_id=UXLOT5FU522U73SMQEUIZ4K43WTQEAV5XRVSGG6EWQON43UWQAYQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.827625579 (604h4m49.698708614s) [noise=true , address=145.79.227.4:4218 , piece_id=2H4DI7772BR4JLLGKJYTSKTKVOABDB4T6UJ74OBTLXHV23WPN5HA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.827942963 (604h4m49.699025998s) [noise=true , address=45.91.116.9:4288 , piece_id=QLRNLFOW2RZ6JLJDNJEKBQ6JWJLOZ57NBPADF3KYVLL4L5FTC62A, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.828225852 (604h4m49.699308887s) [noise=true , address=45.91.116.14:4199 , piece_id=J3LDA3N2Q6B3VTWS5LG25LUD5E2L7I3FD56JPOVJVCJYLXC3AKXQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.828502729 (604h4m49.699585764s) [noise=true , address=64.49.13.138:4483 , piece_id=GRAGOWDG4I5TDHVNX75QJBTS7NIIRY3B73VWDO7DWNITNJXBAYWQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.828558776 (604h4m49.699641821s) [noise=true , address=64.49.13.169:4708 , piece_id=BXCIK6V4R5LZHWUTJOCL66442ZM6T7IYWD2OH5OM2SM7GX5JBIZA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.828853557 (604h4m49.699936592s) [noise=true , address=145.79.227.50:4384 , piece_id=U5R4PT66AXFJFPAGFADCVX6NNNIFLFM3CFTREKVEOOUDHTVTUTBA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.829160732 (604h4m49.700243767s) [noise=true , address=5.231.73.4:4027 , piece_id=DVNFM76WSUGT65ZKJY4RFVXXLZIDQJ7TQ5CYBPUMZ2GXQWJOIY3Q, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.829534374 (604h4m49.700617409s) [noise=true , address=145.79.227.52:4458 , piece_id=2XCSF7MRBBZBGEJ2GE63K25BO5DFB7DUSZ4KXMFRS5NOOHPSEJRA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.829873299 (604h4m49.700956334s) [noise=true , address=145.79.227.34:4908 , piece_id=X57UEZNRHXKLR7R3VEWRKCALEHCMTE7NR73QGALOEZ7NKOBKDDFA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.830200823 (604h4m49.701283868s) [noise=true , address=145.79.227.33:4887 , piece_id=I7CHA6ZOBOS7J4SHID6MQLGAVR66BZ2D2J4VSTFCQW525BPN5KHQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.838625865 (604h4m49.7097089s) [noise=true , address=145.79.227.3:4180 , piece_id=CYOSWRJR37YSHNJBWZZPMOLM7TVRJ2GFEV4XNNHRAQJWFU6Y2JIQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.839781686 (604h4m49.710864731s) [noise=true , address=145.79.227.36:4546 , piece_id=4V7GWX3MULB6CR3QH44ZIOISDONXKJ334XFSYHXUECJZBMRHQCKA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.840846864 (604h4m49.711929899s) [noise=true , address=77.90.31.14:4669 , piece_id=O2Z45BXHUPJQZZ7EJG4FKS3MNDPHWPBT4EMZCZV2HTJSEVAI7WHQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.841958571 (604h4m49.713041606s) [noise=true , address=77.90.31.10:4534 , piece_id=KPMW47ZBIINLCZ67FH6EGEDFLEAT7F35CYP62D4TT7IBLQ57I2FA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:51.267701872 (604h4m51.138784917s) [upload=529467]: Next returned nil segment Dec 30 19:33:51.267719867 (604h4m51.138802912s) [upload=529467]: Waiting for error group managing segments... Dec 30 19:33:51.700925266 (604h4m51.572008302s) [noise=true , address=145.79.227.3:4180 , piece_id=CYOSWRJR37YSHNJBWZZPMOLM7TVRJ2GFEV4XNNHRAQJWFU6Y2JIQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.707415294 (604h4m51.578498329s) [noise=true , address=145.79.227.50:4384 , piece_id=U5R4PT66AXFJFPAGFADCVX6NNNIFLFM3CFTREKVEOOUDHTVTUTBA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.732565163 (604h4m51.603648198s) [noise=true , address=145.79.227.36:4546 , piece_id=4V7GWX3MULB6CR3QH44ZIOISDONXKJ334XFSYHXUECJZBMRHQCKA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.745717487 (604h4m51.616800522s) [noise=true , address=64.49.13.170:4717 , piece_id=NSYCFF26VLJ6QPF5JAPBZRBLULLDAWUEPHI5HZTPSXQVX43KQHVQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.782268034 (604h4m51.653351069s) [noise=true , address=64.49.13.138:4483 , piece_id=GRAGOWDG4I5TDHVNX75QJBTS7NIIRY3B73VWDO7DWNITNJXBAYWQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.856613327 (604h4m51.727696362s) [noise=true , address=145.79.227.52:4458 , piece_id=2XCSF7MRBBZBGEJ2GE63K25BO5DFB7DUSZ4KXMFRS5NOOHPSEJRA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.900758014 (604h4m51.771841059s) [noise=true , address=77.90.31.10:4534 , piece_id=KPMW47ZBIINLCZ67FH6EGEDFLEAT7F35CYP62D4TT7IBLQ57I2FA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.968806577 (604h4m51.839889622s) [noise=true , address=77.90.31.13:4631 , piece_id=IDYOBFP732AUWTYULX2I5ETBTS3DFAVW2EVEZ4BB5NYZFA4P2FTA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.000820185 (604h4m51.87190323s) [noise=true , address=45.91.116.14:4199 , piece_id=J3LDA3N2Q6B3VTWS5LG25LUD5E2L7I3FD56JPOVJVCJYLXC3AKXQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.111919172 (604h4m51.983002207s) [noise=true , address=145.79.227.34:4908 , piece_id=X57UEZNRHXKLR7R3VEWRKCALEHCMTE7NR73QGALOEZ7NKOBKDDFA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.513345511 (604h4m52.384428556s) [noise=true , address=5.231.73.4:4027 , piece_id=DVNFM76WSUGT65ZKJY4RFVXXLZIDQJ7TQ5CYBPUMZ2GXQWJOIY3Q, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.525239208 (604h4m52.396322253s) [noise=true , address=145.79.227.49:4362 , piece_id=UXLOT5FU522U73SMQEUIZ4K43WTQEAV5XRVSGG6EWQON43UWQAYQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.547688835 (604h4m52.41877187s) [noise=true , address=145.79.227.30:4793 , piece_id=37HWPUMM2Q4CZDYKXZIKL5KKPEH4O6ASHI675HEP2K7EOCGBVLFA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.777085496 (604h4m52.648168531s) [noise=true , address=145.79.227.33:4887 , piece_id=I7CHA6ZOBOS7J4SHID6MQLGAVR66BZ2D2J4VSTFCQW525BPN5KHQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.777110584 (604h4m52.648193619s) [seg_pos={0 0}, upload=529467]: Segment reached optimal threshold of 14 pieces. Dec 30 19:33:52.777320053 (604h4m52.648403088s) [noise=true , address=45.91.116.9:4288 , piece_id=QLRNLFOW2RZ6JLJDNJEKBQ6JWJLOZ57NBPADF3KYVLL4L5FTC62A, seg_pos={0 0}, upload=529467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX): protocol: write tcp 188.166.225.170:38490->45.91.116.9:4288: use of closed network connection; write tcp 188.166.225.170:38490->45.91.116.9:4288: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38490->45.91.116.9:4288: use of closed network connection Dec 30 19:33:52.777326334 (604h4m52.64840938s) [noise=true , address=45.91.116.9:4288 , piece_id=QLRNLFOW2RZ6JLJDNJEKBQ6JWJLOZ57NBPADF3KYVLL4L5FTC62A, seg_pos={0 0}, upload=529467]: Long tail cancellation due to optimal threshold Dec 30 19:33:52.777615194 (604h4m52.648698229s) [noise=true , address=77.90.31.14:4669 , piece_id=O2Z45BXHUPJQZZ7EJG4FKS3MNDPHWPBT4EMZCZV2HTJSEVAI7WHQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T): protocol: expected piece hash; context canceled; EOF Dec 30 19:33:52.777622820 (604h4m52.648705855s) [noise=true , address=77.90.31.14:4669 , piece_id=O2Z45BXHUPJQZZ7EJG4FKS3MNDPHWPBT4EMZCZV2HTJSEVAI7WHQ, seg_pos={0 0}, upload=529467]: Long tail cancellation due to optimal threshold Dec 30 19:33:52.777648788 (604h4m52.648731823s) [noise=true , address=64.49.13.169:4708 , piece_id=BXCIK6V4R5LZHWUTJOCL66442ZM6T7IYWD2OH5OM2SM7GX5JBIZA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:36148->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:36148->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36148->64.49.13.169:4708: use of closed network connection Dec 30 19:33:52.777665260 (604h4m52.648748295s) [noise=true , address=64.49.13.169:4708 , piece_id=BXCIK6V4R5LZHWUTJOCL66442ZM6T7IYWD2OH5OM2SM7GX5JBIZA, seg_pos={0 0}, upload=529467]: Long tail cancellation due to optimal threshold Dec 30 19:33:52.777872405 (604h4m52.64895545s) [noise=true , address=145.79.227.4:4218 , piece_id=2H4DI7772BR4JLLGKJYTSKTKVOABDB4T6UJ74OBTLXHV23WPN5HA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mbzxg1BjmbXeHxGgheZmQoN91k8NgaAUUASPkJe3fEmS4eEkc): protocol: expected piece hash; context canceled; EOF Dec 30 19:33:52.777879939 (604h4m52.648962984s) [noise=true , address=145.79.227.4:4218 , piece_id=2H4DI7772BR4JLLGKJYTSKTKVOABDB4T6UJ74OBTLXHV23WPN5HA, seg_pos={0 0}, upload=529467]: Long tail cancellation due to optimal threshold Dec 30 19:33:52.777898815 (604h4m52.64898185s) [seg_pos={0 0}, upload=529467]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 19:33:52.777974709 (604h4m52.649057755s) [upload=529467]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:33:52.777979669 (604h4m52.649062714s) [upload=529467]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:33:53.082585408 (604h4m52.953668453s) [upload=529467]: Done uploading segments... Dec 30 19:33:53.082605185 (604h4m52.953688221s) [upload=529467]: Upload finished. err: <nil> Duration: 26.676275609s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.X..X.....X..XX...X..X.....XX.X.....X.....X.................] duration: 15.310646203s trace: https://jaeger.dev.storj.io/trace/4b57a4d55c27f17d FAIL (2025-12-30 19:49:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 2MXNCFG5BZSBRLTW4OPJXJWAVDPMHK4GXYPWT6GBFIH65F3IEMVA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: UU2AYD5D3SDGOW7JMKI6CV2SNPTVZNKJUW4TFNWSQ22WH4LAOYTQ) context deadline exceeded ======LOGS====== Dec 30 19:37:18.017831880 (604h8m17.888914925s) [upload=529518]: Starting upload Dec 30 19:37:18.017920869 (604h8m17.889003914s) [upload=529518]: Done starting upload Dec 30 19:37:18.017989800 (604h8m17.889072835s) [upload=529518]: Uploading segments... Dec 30 19:37:18.018167178 (604h8m17.889250213s) [upload=529518]: Got next segment. Inline: false Dec 30 19:37:18.018211322 (604h8m17.889294367s) [upload=529518]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 19:37:18.018216993 (604h8m17.889300028s) [upload=529518]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 19:37:18.265194540 (604h8m18.136277575s) [seg_pos={0 0}, upload=529518]: Begin upload segment... Dec 30 19:37:18.265242221 (604h8m18.136325267s) [seg_pos={0 0}, upload=529518]: Done begin upload segment. Dec 30 19:37:18.265271887 (604h8m18.136354922s) [noise=true , address=45.91.116.13:4156 , piece_id=ZXMVGKUW2Q4SQNBGD5ZZP7CHF2RFVVLSZIYEMVOOZB2VU4CEU6UQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.265280745 (604h8m18.13636378s) [noise=true , address=145.79.227.22:4510 , piece_id=TFAYL6XXPFSLYYPUK5GRBO7FDDDAJRMPGI2PTD3SH7OLW6XR5QGQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.265996978 (604h8m18.137080013s) [noise=true , address=145.79.227.13:4803 , piece_id=DGVTZAR4T33Z4ZWIRBWKLPTY45W7CDF5SPJ7IOSUBFLNQWAD4XOQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.266478024 (604h8m18.137561059s) [noise=true , address=45.91.116.8:4048 , piece_id=2MXNCFG5BZSBRLTW4OPJXJWAVDPMHK4GXYPWT6GBFIH65F3IEMVA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.267063399 (604h8m18.138146444s) [noise=true , address=145.79.227.12:4765 , piece_id=D4M33JNPTNF5EOTP6V4CH2KBZOJBKIRZ5RCEUMJMFSB5KEO3EITA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.267602044 (604h8m18.138685079s) [noise=true , address=45.91.116.11:4118 , piece_id=UU2AYD5D3SDGOW7JMKI6CV2SNPTVZNKJUW4TFNWSQ22WH4LAOYTQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.268103920 (604h8m18.139186965s) [noise=true , address=145.79.227.24:4582 , piece_id=DWQXXGUEQ4IXJJXA57AIULGBQTCM5LMRS6IKSDKNVDXIBAS3HMRA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.268624721 (604h8m18.139707756s) [noise=true , address=145.79.227.2:4114 , piece_id=5ELY6KB4QBNBOPMG6ALNYRCW3CYXRYVSDGMRAJYYYPE6TZSMA4OA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.269149150 (604h8m18.140232195s) [noise=true , address=45.91.116.7:4012 , piece_id=LOW2MEEFBV64QEH7MKQT2R4PHUBPWPRZGDYVCYMELUD4BOQTGEVA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.269680512 (604h8m18.140763547s) [noise=true , address=77.90.31.12:4589 , piece_id=RQH5BFFWHPXUA4NMBZUJYNOAI5OQU43HDCGKJMYBSS6O3YDFQL2A, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.270313487 (604h8m18.141396522s) [noise=true , address=145.79.227.46:4601 , piece_id=Y4BKLMSLP2SBMSKC3S6ISJCIRAXTZMIUXQDTRYXBRIEE6V4YQYLA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.270669034 (604h8m18.141752069s) [noise=true , address=45.91.116.12:4227 , piece_id=Q26IH3AZJJISAEL5LPYT7CRFTP4KUMXOX44MYSG3BCLZNLMYLERQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.270947995 (604h8m18.14203103s) [noise=true , address=145.79.227.23:4547 , piece_id=6G7N6QNXE3HHSWO6BMDRCQS26RJKV7PETS3VCAPLPCJNFHR4WGEA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.271235693 (604h8m18.142318738s) [noise=true , address=77.90.31.21:4880 , piece_id=2EIJYMPCZP4BWQX4I3RSQS5IDOEAMX7UAGQIXEVRJQG6MPBZSFQA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.271512129 (604h8m18.142595174s) [noise=true , address=145.79.227.51:4410 , piece_id=65NLCMFGMV2NLVDAYRXQMYVQDBAWQT4TUAFRIYDUZOGT46S5KYGA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.271809736 (604h8m18.142892782s) [noise=true , address=145.79.227.29:4941 , piece_id=J2QTCA233FRN5E6KQXXIP767JHUL46VQJ7I76UTJEWVLO4UYAHFQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:19.968123940 (604h8m19.839206985s) [upload=529518]: Next returned nil segment Dec 30 19:37:19.968138839 (604h8m19.839221874s) [upload=529518]: Waiting for error group managing segments... Dec 30 19:37:20.240602244 (604h8m20.111685279s) [noise=true , address=145.79.227.13:4803 , piece_id=DGVTZAR4T33Z4ZWIRBWKLPTY45W7CDF5SPJ7IOSUBFLNQWAD4XOQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.242668559 (604h8m20.113751594s) [noise=true , address=145.79.227.2:4114 , piece_id=5ELY6KB4QBNBOPMG6ALNYRCW3CYXRYVSDGMRAJYYYPE6TZSMA4OA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.274323484 (604h8m20.145406529s) [noise=true , address=145.79.227.51:4410 , piece_id=65NLCMFGMV2NLVDAYRXQMYVQDBAWQT4TUAFRIYDUZOGT46S5KYGA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.276151625 (604h8m20.14723467s) [noise=true , address=145.79.227.24:4582 , piece_id=DWQXXGUEQ4IXJJXA57AIULGBQTCM5LMRS6IKSDKNVDXIBAS3HMRA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.325870757 (604h8m20.196953792s) [noise=true , address=145.79.227.46:4601 , piece_id=Y4BKLMSLP2SBMSKC3S6ISJCIRAXTZMIUXQDTRYXBRIEE6V4YQYLA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.338603552 (604h8m20.209686587s) [noise=true , address=145.79.227.22:4510 , piece_id=TFAYL6XXPFSLYYPUK5GRBO7FDDDAJRMPGI2PTD3SH7OLW6XR5QGQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.339426289 (604h8m20.210509334s) [noise=true , address=145.79.227.12:4765 , piece_id=D4M33JNPTNF5EOTP6V4CH2KBZOJBKIRZ5RCEUMJMFSB5KEO3EITA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.528772708 (604h8m20.399855963s) [noise=true , address=77.90.31.21:4880 , piece_id=2EIJYMPCZP4BWQX4I3RSQS5IDOEAMX7UAGQIXEVRJQG6MPBZSFQA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.723971783 (604h8m20.595054818s) [noise=true , address=45.91.116.12:4227 , piece_id=Q26IH3AZJJISAEL5LPYT7CRFTP4KUMXOX44MYSG3BCLZNLMYLERQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.738010033 (604h8m20.609093068s) [noise=true , address=45.91.116.7:4012 , piece_id=LOW2MEEFBV64QEH7MKQT2R4PHUBPWPRZGDYVCYMELUD4BOQTGEVA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.900656063 (604h8m20.771739098s) [noise=true , address=45.91.116.8:4048 , piece_id=2MXNCFG5BZSBRLTW4OPJXJWAVDPMHK4GXYPWT6GBFIH65F3IEMVA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:21.200841094 (604h8m21.07192414s) [noise=true , address=77.90.31.12:4589 , piece_id=RQH5BFFWHPXUA4NMBZUJYNOAI5OQU43HDCGKJMYBSS6O3YDFQL2A, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:21.316516896 (604h8m21.187599941s) [noise=true , address=45.91.116.11:4118 , piece_id=UU2AYD5D3SDGOW7JMKI6CV2SNPTVZNKJUW4TFNWSQ22WH4LAOYTQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:21.316538046 (604h8m21.187621081s) [seg_pos={0 0}, upload=529518]: Segment reached optimal threshold of 13 pieces. Dec 30 19:37:21.316716626 (604h8m21.187799661s) [noise=true , address=145.79.227.29:4941 , piece_id=J2QTCA233FRN5E6KQXXIP767JHUL46VQJ7I76UTJEWVLO4UYAHFQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 30 19:37:21.316723229 (604h8m21.187806264s) [noise=true , address=145.79.227.29:4941 , piece_id=J2QTCA233FRN5E6KQXXIP767JHUL46VQJ7I76UTJEWVLO4UYAHFQ, seg_pos={0 0}, upload=529518]: Long tail cancellation due to optimal threshold Dec 30 19:37:21.316823179 (604h8m21.187906214s) [noise=true , address=45.91.116.13:4156 , piece_id=ZXMVGKUW2Q4SQNBGD5ZZP7CHF2RFVVLSZIYEMVOOZB2VU4CEU6UQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 30 19:37:21.316831525 (604h8m21.18791456s) [noise=true , address=45.91.116.13:4156 , piece_id=ZXMVGKUW2Q4SQNBGD5ZZP7CHF2RFVVLSZIYEMVOOZB2VU4CEU6UQ, seg_pos={0 0}, upload=529518]: Long tail cancellation due to optimal threshold Dec 30 19:37:21.317004414 (604h8m21.18808745s) [noise=true , address=145.79.227.23:4547 , piece_id=6G7N6QNXE3HHSWO6BMDRCQS26RJKV7PETS3VCAPLPCJNFHR4WGEA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: write tcp 188.166.225.170:45998->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:45998->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45998->145.79.227.23:4547: use of closed network connection Dec 30 19:37:21.317013822 (604h8m21.188096877s) [noise=true , address=145.79.227.23:4547 , piece_id=6G7N6QNXE3HHSWO6BMDRCQS26RJKV7PETS3VCAPLPCJNFHR4WGEA, seg_pos={0 0}, upload=529518]: Long tail cancellation due to optimal threshold Dec 30 19:37:21.317033068 (604h8m21.188116113s) [seg_pos={0 0}, upload=529518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 19:37:21.317121227 (604h8m21.188204262s) [upload=529518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:37:21.317128490 (604h8m21.188211555s) [upload=529518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:37:21.616086595 (604h8m21.48716965s) [upload=529518]: Done uploading segments... Dec 30 19:37:21.616243293 (604h8m21.487326328s) [upload=529518]: Upload finished. err: <nil> Duration: 15.31067127s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X..X.....................X....................] duration: 38.935733476s trace: https://jaeger.dev.storj.io/trace/2e4bd3c87074131f FAIL (2025-12-30 19:49:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== 98432889s) [noise=true , address=145.79.227.49:4362 , piece_id=UXLOT5FU522U73SMQEUIZ4K43WTQEAV5XRVSGG6EWQON43UWQAYQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.827625579 (604h4m49.698708614s) [noise=true , address=145.79.227.4:4218 , piece_id=2H4DI7772BR4JLLGKJYTSKTKVOABDB4T6UJ74OBTLXHV23WPN5HA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.827942963 (604h4m49.699025998s) [noise=true , address=45.91.116.9:4288 , piece_id=QLRNLFOW2RZ6JLJDNJEKBQ6JWJLOZ57NBPADF3KYVLL4L5FTC62A, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.828225852 (604h4m49.699308887s) [noise=true , address=45.91.116.14:4199 , piece_id=J3LDA3N2Q6B3VTWS5LG25LUD5E2L7I3FD56JPOVJVCJYLXC3AKXQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.828502729 (604h4m49.699585764s) [noise=true , address=64.49.13.138:4483 , piece_id=GRAGOWDG4I5TDHVNX75QJBTS7NIIRY3B73VWDO7DWNITNJXBAYWQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.828558776 (604h4m49.699641821s) [noise=true , address=64.49.13.169:4708 , piece_id=BXCIK6V4R5LZHWUTJOCL66442ZM6T7IYWD2OH5OM2SM7GX5JBIZA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.828853557 (604h4m49.699936592s) [noise=true , address=145.79.227.50:4384 , piece_id=U5R4PT66AXFJFPAGFADCVX6NNNIFLFM3CFTREKVEOOUDHTVTUTBA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.829160732 (604h4m49.700243767s) [noise=true , address=5.231.73.4:4027 , piece_id=DVNFM76WSUGT65ZKJY4RFVXXLZIDQJ7TQ5CYBPUMZ2GXQWJOIY3Q, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.829534374 (604h4m49.700617409s) [noise=true , address=145.79.227.52:4458 , piece_id=2XCSF7MRBBZBGEJ2GE63K25BO5DFB7DUSZ4KXMFRS5NOOHPSEJRA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.829873299 (604h4m49.700956334s) [noise=true , address=145.79.227.34:4908 , piece_id=X57UEZNRHXKLR7R3VEWRKCALEHCMTE7NR73QGALOEZ7NKOBKDDFA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.830200823 (604h4m49.701283868s) [noise=true , address=145.79.227.33:4887 , piece_id=I7CHA6ZOBOS7J4SHID6MQLGAVR66BZ2D2J4VSTFCQW525BPN5KHQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.838625865 (604h4m49.7097089s) [noise=true , address=145.79.227.3:4180 , piece_id=CYOSWRJR37YSHNJBWZZPMOLM7TVRJ2GFEV4XNNHRAQJWFU6Y2JIQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.839781686 (604h4m49.710864731s) [noise=true , address=145.79.227.36:4546 , piece_id=4V7GWX3MULB6CR3QH44ZIOISDONXKJ334XFSYHXUECJZBMRHQCKA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.840846864 (604h4m49.711929899s) [noise=true , address=77.90.31.14:4669 , piece_id=O2Z45BXHUPJQZZ7EJG4FKS3MNDPHWPBT4EMZCZV2HTJSEVAI7WHQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.841958571 (604h4m49.713041606s) [noise=true , address=77.90.31.10:4534 , piece_id=KPMW47ZBIINLCZ67FH6EGEDFLEAT7F35CYP62D4TT7IBLQ57I2FA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:51.267701872 (604h4m51.138784917s) [upload=529467]: Next returned nil segment Dec 30 19:33:51.267719867 (604h4m51.138802912s) [upload=529467]: Waiting for error group managing segments... Dec 30 19:33:51.700925266 (604h4m51.572008302s) [noise=true , address=145.79.227.3:4180 , piece_id=CYOSWRJR37YSHNJBWZZPMOLM7TVRJ2GFEV4XNNHRAQJWFU6Y2JIQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.707415294 (604h4m51.578498329s) [noise=true , address=145.79.227.50:4384 , piece_id=U5R4PT66AXFJFPAGFADCVX6NNNIFLFM3CFTREKVEOOUDHTVTUTBA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.732565163 (604h4m51.603648198s) [noise=true , address=145.79.227.36:4546 , piece_id=4V7GWX3MULB6CR3QH44ZIOISDONXKJ334XFSYHXUECJZBMRHQCKA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.745717487 (604h4m51.616800522s) [noise=true , address=64.49.13.170:4717 , piece_id=NSYCFF26VLJ6QPF5JAPBZRBLULLDAWUEPHI5HZTPSXQVX43KQHVQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.782268034 (604h4m51.653351069s) [noise=true , address=64.49.13.138:4483 , piece_id=GRAGOWDG4I5TDHVNX75QJBTS7NIIRY3B73VWDO7DWNITNJXBAYWQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.856613327 (604h4m51.727696362s) [noise=true , address=145.79.227.52:4458 , piece_id=2XCSF7MRBBZBGEJ2GE63K25BO5DFB7DUSZ4KXMFRS5NOOHPSEJRA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.900758014 (604h4m51.771841059s) [noise=true , address=77.90.31.10:4534 , piece_id=KPMW47ZBIINLCZ67FH6EGEDFLEAT7F35CYP62D4TT7IBLQ57I2FA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.968806577 (604h4m51.839889622s) [noise=true , address=77.90.31.13:4631 , piece_id=IDYOBFP732AUWTYULX2I5ETBTS3DFAVW2EVEZ4BB5NYZFA4P2FTA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.000820185 (604h4m51.87190323s) [noise=true , address=45.91.116.14:4199 , piece_id=J3LDA3N2Q6B3VTWS5LG25LUD5E2L7I3FD56JPOVJVCJYLXC3AKXQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.111919172 (604h4m51.983002207s) [noise=true , address=145.79.227.34:4908 , piece_id=X57UEZNRHXKLR7R3VEWRKCALEHCMTE7NR73QGALOEZ7NKOBKDDFA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.513345511 (604h4m52.384428556s) [noise=true , address=5.231.73.4:4027 , piece_id=DVNFM76WSUGT65ZKJY4RFVXXLZIDQJ7TQ5CYBPUMZ2GXQWJOIY3Q, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.525239208 (604h4m52.396322253s) [noise=true , address=145.79.227.49:4362 , piece_id=UXLOT5FU522U73SMQEUIZ4K43WTQEAV5XRVSGG6EWQON43UWQAYQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.547688835 (604h4m52.41877187s) [noise=true , address=145.79.227.30:4793 , piece_id=37HWPUMM2Q4CZDYKXZIKL5KKPEH4O6ASHI675HEP2K7EOCGBVLFA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.777085496 (604h4m52.648168531s) [noise=true , address=145.79.227.33:4887 , piece_id=I7CHA6ZOBOS7J4SHID6MQLGAVR66BZ2D2J4VSTFCQW525BPN5KHQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.777110584 (604h4m52.648193619s) [seg_pos={0 0}, upload=529467]: Segment reached optimal threshold of 14 pieces. Dec 30 19:33:52.777320053 (604h4m52.648403088s) [noise=true , address=45.91.116.9:4288 , piece_id=QLRNLFOW2RZ6JLJDNJEKBQ6JWJLOZ57NBPADF3KYVLL4L5FTC62A, seg_pos={0 0}, upload=529467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX): protocol: write tcp 188.166.225.170:38490->45.91.116.9:4288: use of closed network connection; write tcp 188.166.225.170:38490->45.91.116.9:4288: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38490->45.91.116.9:4288: use of closed network connection Dec 30 19:33:52.777326334 (604h4m52.64840938s) [noise=true , address=45.91.116.9:4288 , piece_id=QLRNLFOW2RZ6JLJDNJEKBQ6JWJLOZ57NBPADF3KYVLL4L5FTC62A, seg_pos={0 0}, upload=529467]: Long tail cancellation due to optimal threshold Dec 30 19:33:52.777615194 (604h4m52.648698229s) [noise=true , address=77.90.31.14:4669 , piece_id=O2Z45BXHUPJQZZ7EJG4FKS3MNDPHWPBT4EMZCZV2HTJSEVAI7WHQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T): protocol: expected piece hash; context canceled; EOF Dec 30 19:33:52.777622820 (604h4m52.648705855s) [noise=true , address=77.90.31.14:4669 , piece_id=O2Z45BXHUPJQZZ7EJG4FKS3MNDPHWPBT4EMZCZV2HTJSEVAI7WHQ, seg_pos={0 0}, upload=529467]: Long tail cancellation due to optimal threshold Dec 30 19:33:52.777648788 (604h4m52.648731823s) [noise=true , address=64.49.13.169:4708 , piece_id=BXCIK6V4R5LZHWUTJOCL66442ZM6T7IYWD2OH5OM2SM7GX5JBIZA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:36148->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:36148->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36148->64.49.13.169:4708: use of closed network connection Dec 30 19:33:52.777665260 (604h4m52.648748295s) [noise=true , address=64.49.13.169:4708 , piece_id=BXCIK6V4R5LZHWUTJOCL66442ZM6T7IYWD2OH5OM2SM7GX5JBIZA, seg_pos={0 0}, upload=529467]: Long tail cancellation due to optimal threshold Dec 30 19:33:52.777872405 (604h4m52.64895545s) [noise=true , address=145.79.227.4:4218 , piece_id=2H4DI7772BR4JLLGKJYTSKTKVOABDB4T6UJ74OBTLXHV23WPN5HA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mbzxg1BjmbXeHxGgheZmQoN91k8NgaAUUASPkJe3fEmS4eEkc): protocol: expected piece hash; context canceled; EOF Dec 30 19:33:52.777879939 (604h4m52.648962984s) [noise=true , address=145.79.227.4:4218 , piece_id=2H4DI7772BR4JLLGKJYTSKTKVOABDB4T6UJ74OBTLXHV23WPN5HA, seg_pos={0 0}, upload=529467]: Long tail cancellation due to optimal threshold Dec 30 19:33:52.777898815 (604h4m52.64898185s) [seg_pos={0 0}, upload=529467]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 19:33:52.777974709 (604h4m52.649057755s) [upload=529467]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:33:52.777979669 (604h4m52.649062714s) [upload=529467]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:33:53.082585408 (604h4m52.953668453s) [upload=529467]: Done uploading segments... Dec 30 19:33:53.082605185 (604h4m52.953688221s) [upload=529467]: Upload finished. err: <nil> Duration: 38.935767489s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..X..X.....X..XX...X..X.....XX.X.....X.....X................] duration: 5.95596532s trace: https://jaeger.dev.storj.io/trace/4b57a4d55c27f17d FAIL (2025-12-30 19:49:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 2MXNCFG5BZSBRLTW4OPJXJWAVDPMHK4GXYPWT6GBFIH65F3IEMVA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: UU2AYD5D3SDGOW7JMKI6CV2SNPTVZNKJUW4TFNWSQ22WH4LAOYTQ) context deadline exceeded ======LOGS====== Dec 30 19:37:18.017831880 (604h8m17.888914925s) [upload=529518]: Starting upload Dec 30 19:37:18.017920869 (604h8m17.889003914s) [upload=529518]: Done starting upload Dec 30 19:37:18.017989800 (604h8m17.889072835s) [upload=529518]: Uploading segments... Dec 30 19:37:18.018167178 (604h8m17.889250213s) [upload=529518]: Got next segment. Inline: false Dec 30 19:37:18.018211322 (604h8m17.889294367s) [upload=529518]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 19:37:18.018216993 (604h8m17.889300028s) [upload=529518]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 19:37:18.265194540 (604h8m18.136277575s) [seg_pos={0 0}, upload=529518]: Begin upload segment... Dec 30 19:37:18.265242221 (604h8m18.136325267s) [seg_pos={0 0}, upload=529518]: Done begin upload segment. Dec 30 19:37:18.265271887 (604h8m18.136354922s) [noise=true , address=45.91.116.13:4156 , piece_id=ZXMVGKUW2Q4SQNBGD5ZZP7CHF2RFVVLSZIYEMVOOZB2VU4CEU6UQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.265280745 (604h8m18.13636378s) [noise=true , address=145.79.227.22:4510 , piece_id=TFAYL6XXPFSLYYPUK5GRBO7FDDDAJRMPGI2PTD3SH7OLW6XR5QGQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.265996978 (604h8m18.137080013s) [noise=true , address=145.79.227.13:4803 , piece_id=DGVTZAR4T33Z4ZWIRBWKLPTY45W7CDF5SPJ7IOSUBFLNQWAD4XOQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.266478024 (604h8m18.137561059s) [noise=true , address=45.91.116.8:4048 , piece_id=2MXNCFG5BZSBRLTW4OPJXJWAVDPMHK4GXYPWT6GBFIH65F3IEMVA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.267063399 (604h8m18.138146444s) [noise=true , address=145.79.227.12:4765 , piece_id=D4M33JNPTNF5EOTP6V4CH2KBZOJBKIRZ5RCEUMJMFSB5KEO3EITA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.267602044 (604h8m18.138685079s) [noise=true , address=45.91.116.11:4118 , piece_id=UU2AYD5D3SDGOW7JMKI6CV2SNPTVZNKJUW4TFNWSQ22WH4LAOYTQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.268103920 (604h8m18.139186965s) [noise=true , address=145.79.227.24:4582 , piece_id=DWQXXGUEQ4IXJJXA57AIULGBQTCM5LMRS6IKSDKNVDXIBAS3HMRA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.268624721 (604h8m18.139707756s) [noise=true , address=145.79.227.2:4114 , piece_id=5ELY6KB4QBNBOPMG6ALNYRCW3CYXRYVSDGMRAJYYYPE6TZSMA4OA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.269149150 (604h8m18.140232195s) [noise=true , address=45.91.116.7:4012 , piece_id=LOW2MEEFBV64QEH7MKQT2R4PHUBPWPRZGDYVCYMELUD4BOQTGEVA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.269680512 (604h8m18.140763547s) [noise=true , address=77.90.31.12:4589 , piece_id=RQH5BFFWHPXUA4NMBZUJYNOAI5OQU43HDCGKJMYBSS6O3YDFQL2A, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.270313487 (604h8m18.141396522s) [noise=true , address=145.79.227.46:4601 , piece_id=Y4BKLMSLP2SBMSKC3S6ISJCIRAXTZMIUXQDTRYXBRIEE6V4YQYLA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.270669034 (604h8m18.141752069s) [noise=true , address=45.91.116.12:4227 , piece_id=Q26IH3AZJJISAEL5LPYT7CRFTP4KUMXOX44MYSG3BCLZNLMYLERQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.270947995 (604h8m18.14203103s) [noise=true , address=145.79.227.23:4547 , piece_id=6G7N6QNXE3HHSWO6BMDRCQS26RJKV7PETS3VCAPLPCJNFHR4WGEA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.271235693 (604h8m18.142318738s) [noise=true , address=77.90.31.21:4880 , piece_id=2EIJYMPCZP4BWQX4I3RSQS5IDOEAMX7UAGQIXEVRJQG6MPBZSFQA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.271512129 (604h8m18.142595174s) [noise=true , address=145.79.227.51:4410 , piece_id=65NLCMFGMV2NLVDAYRXQMYVQDBAWQT4TUAFRIYDUZOGT46S5KYGA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.271809736 (604h8m18.142892782s) [noise=true , address=145.79.227.29:4941 , piece_id=J2QTCA233FRN5E6KQXXIP767JHUL46VQJ7I76UTJEWVLO4UYAHFQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:19.968123940 (604h8m19.839206985s) [upload=529518]: Next returned nil segment Dec 30 19:37:19.968138839 (604h8m19.839221874s) [upload=529518]: Waiting for error group managing segments... Dec 30 19:37:20.240602244 (604h8m20.111685279s) [noise=true , address=145.79.227.13:4803 , piece_id=DGVTZAR4T33Z4ZWIRBWKLPTY45W7CDF5SPJ7IOSUBFLNQWAD4XOQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.242668559 (604h8m20.113751594s) [noise=true , address=145.79.227.2:4114 , piece_id=5ELY6KB4QBNBOPMG6ALNYRCW3CYXRYVSDGMRAJYYYPE6TZSMA4OA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.274323484 (604h8m20.145406529s) [noise=true , address=145.79.227.51:4410 , piece_id=65NLCMFGMV2NLVDAYRXQMYVQDBAWQT4TUAFRIYDUZOGT46S5KYGA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.276151625 (604h8m20.14723467s) [noise=true , address=145.79.227.24:4582 , piece_id=DWQXXGUEQ4IXJJXA57AIULGBQTCM5LMRS6IKSDKNVDXIBAS3HMRA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.325870757 (604h8m20.196953792s) [noise=true , address=145.79.227.46:4601 , piece_id=Y4BKLMSLP2SBMSKC3S6ISJCIRAXTZMIUXQDTRYXBRIEE6V4YQYLA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.338603552 (604h8m20.209686587s) [noise=true , address=145.79.227.22:4510 , piece_id=TFAYL6XXPFSLYYPUK5GRBO7FDDDAJRMPGI2PTD3SH7OLW6XR5QGQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.339426289 (604h8m20.210509334s) [noise=true , address=145.79.227.12:4765 , piece_id=D4M33JNPTNF5EOTP6V4CH2KBZOJBKIRZ5RCEUMJMFSB5KEO3EITA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.528772708 (604h8m20.399855963s) [noise=true , address=77.90.31.21:4880 , piece_id=2EIJYMPCZP4BWQX4I3RSQS5IDOEAMX7UAGQIXEVRJQG6MPBZSFQA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.723971783 (604h8m20.595054818s) [noise=true , address=45.91.116.12:4227 , piece_id=Q26IH3AZJJISAEL5LPYT7CRFTP4KUMXOX44MYSG3BCLZNLMYLERQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.738010033 (604h8m20.609093068s) [noise=true , address=45.91.116.7:4012 , piece_id=LOW2MEEFBV64QEH7MKQT2R4PHUBPWPRZGDYVCYMELUD4BOQTGEVA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.900656063 (604h8m20.771739098s) [noise=true , address=45.91.116.8:4048 , piece_id=2MXNCFG5BZSBRLTW4OPJXJWAVDPMHK4GXYPWT6GBFIH65F3IEMVA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:21.200841094 (604h8m21.07192414s) [noise=true , address=77.90.31.12:4589 , piece_id=RQH5BFFWHPXUA4NMBZUJYNOAI5OQU43HDCGKJMYBSS6O3YDFQL2A, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:21.316516896 (604h8m21.187599941s) [noise=true , address=45.91.116.11:4118 , piece_id=UU2AYD5D3SDGOW7JMKI6CV2SNPTVZNKJUW4TFNWSQ22WH4LAOYTQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:21.316538046 (604h8m21.187621081s) [seg_pos={0 0}, upload=529518]: Segment reached optimal threshold of 13 pieces. Dec 30 19:37:21.316716626 (604h8m21.187799661s) [noise=true , address=145.79.227.29:4941 , piece_id=J2QTCA233FRN5E6KQXXIP767JHUL46VQJ7I76UTJEWVLO4UYAHFQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 30 19:37:21.316723229 (604h8m21.187806264s) [noise=true , address=145.79.227.29:4941 , piece_id=J2QTCA233FRN5E6KQXXIP767JHUL46VQJ7I76UTJEWVLO4UYAHFQ, seg_pos={0 0}, upload=529518]: Long tail cancellation due to optimal threshold Dec 30 19:37:21.316823179 (604h8m21.187906214s) [noise=true , address=45.91.116.13:4156 , piece_id=ZXMVGKUW2Q4SQNBGD5ZZP7CHF2RFVVLSZIYEMVOOZB2VU4CEU6UQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 30 19:37:21.316831525 (604h8m21.18791456s) [noise=true , address=45.91.116.13:4156 , piece_id=ZXMVGKUW2Q4SQNBGD5ZZP7CHF2RFVVLSZIYEMVOOZB2VU4CEU6UQ, seg_pos={0 0}, upload=529518]: Long tail cancellation due to optimal threshold Dec 30 19:37:21.317004414 (604h8m21.18808745s) [noise=true , address=145.79.227.23:4547 , piece_id=6G7N6QNXE3HHSWO6BMDRCQS26RJKV7PETS3VCAPLPCJNFHR4WGEA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: write tcp 188.166.225.170:45998->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:45998->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45998->145.79.227.23:4547: use of closed network connection Dec 30 19:37:21.317013822 (604h8m21.188096877s) [noise=true , address=145.79.227.23:4547 , piece_id=6G7N6QNXE3HHSWO6BMDRCQS26RJKV7PETS3VCAPLPCJNFHR4WGEA, seg_pos={0 0}, upload=529518]: Long tail cancellation due to optimal threshold Dec 30 19:37:21.317033068 (604h8m21.188116113s) [seg_pos={0 0}, upload=529518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 19:37:21.317121227 (604h8m21.188204262s) [upload=529518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:37:21.317128490 (604h8m21.188211555s) [upload=529518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:37:21.616086595 (604h8m21.48716965s) [upload=529518]: Done uploading segments... Dec 30 19:37:21.616243293 (604h8m21.487326328s) [upload=529518]: Upload finished. err: <nil> Duration: 5.956001858s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X..X.....................X...................] duration: 37.95100118s trace: https://jaeger.dev.storj.io/trace/2e4bd3c87074131f FAIL (2025-12-30 19:48:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== 98432889s) [noise=true , address=145.79.227.49:4362 , piece_id=UXLOT5FU522U73SMQEUIZ4K43WTQEAV5XRVSGG6EWQON43UWQAYQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.827625579 (604h4m49.698708614s) [noise=true , address=145.79.227.4:4218 , piece_id=2H4DI7772BR4JLLGKJYTSKTKVOABDB4T6UJ74OBTLXHV23WPN5HA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.827942963 (604h4m49.699025998s) [noise=true , address=45.91.116.9:4288 , piece_id=QLRNLFOW2RZ6JLJDNJEKBQ6JWJLOZ57NBPADF3KYVLL4L5FTC62A, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.828225852 (604h4m49.699308887s) [noise=true , address=45.91.116.14:4199 , piece_id=J3LDA3N2Q6B3VTWS5LG25LUD5E2L7I3FD56JPOVJVCJYLXC3AKXQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.828502729 (604h4m49.699585764s) [noise=true , address=64.49.13.138:4483 , piece_id=GRAGOWDG4I5TDHVNX75QJBTS7NIIRY3B73VWDO7DWNITNJXBAYWQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.828558776 (604h4m49.699641821s) [noise=true , address=64.49.13.169:4708 , piece_id=BXCIK6V4R5LZHWUTJOCL66442ZM6T7IYWD2OH5OM2SM7GX5JBIZA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.828853557 (604h4m49.699936592s) [noise=true , address=145.79.227.50:4384 , piece_id=U5R4PT66AXFJFPAGFADCVX6NNNIFLFM3CFTREKVEOOUDHTVTUTBA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.829160732 (604h4m49.700243767s) [noise=true , address=5.231.73.4:4027 , piece_id=DVNFM76WSUGT65ZKJY4RFVXXLZIDQJ7TQ5CYBPUMZ2GXQWJOIY3Q, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.829534374 (604h4m49.700617409s) [noise=true , address=145.79.227.52:4458 , piece_id=2XCSF7MRBBZBGEJ2GE63K25BO5DFB7DUSZ4KXMFRS5NOOHPSEJRA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.829873299 (604h4m49.700956334s) [noise=true , address=145.79.227.34:4908 , piece_id=X57UEZNRHXKLR7R3VEWRKCALEHCMTE7NR73QGALOEZ7NKOBKDDFA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.830200823 (604h4m49.701283868s) [noise=true , address=145.79.227.33:4887 , piece_id=I7CHA6ZOBOS7J4SHID6MQLGAVR66BZ2D2J4VSTFCQW525BPN5KHQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.838625865 (604h4m49.7097089s) [noise=true , address=145.79.227.3:4180 , piece_id=CYOSWRJR37YSHNJBWZZPMOLM7TVRJ2GFEV4XNNHRAQJWFU6Y2JIQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.839781686 (604h4m49.710864731s) [noise=true , address=145.79.227.36:4546 , piece_id=4V7GWX3MULB6CR3QH44ZIOISDONXKJ334XFSYHXUECJZBMRHQCKA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.840846864 (604h4m49.711929899s) [noise=true , address=77.90.31.14:4669 , piece_id=O2Z45BXHUPJQZZ7EJG4FKS3MNDPHWPBT4EMZCZV2HTJSEVAI7WHQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.841958571 (604h4m49.713041606s) [noise=true , address=77.90.31.10:4534 , piece_id=KPMW47ZBIINLCZ67FH6EGEDFLEAT7F35CYP62D4TT7IBLQ57I2FA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:51.267701872 (604h4m51.138784917s) [upload=529467]: Next returned nil segment Dec 30 19:33:51.267719867 (604h4m51.138802912s) [upload=529467]: Waiting for error group managing segments... Dec 30 19:33:51.700925266 (604h4m51.572008302s) [noise=true , address=145.79.227.3:4180 , piece_id=CYOSWRJR37YSHNJBWZZPMOLM7TVRJ2GFEV4XNNHRAQJWFU6Y2JIQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.707415294 (604h4m51.578498329s) [noise=true , address=145.79.227.50:4384 , piece_id=U5R4PT66AXFJFPAGFADCVX6NNNIFLFM3CFTREKVEOOUDHTVTUTBA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.732565163 (604h4m51.603648198s) [noise=true , address=145.79.227.36:4546 , piece_id=4V7GWX3MULB6CR3QH44ZIOISDONXKJ334XFSYHXUECJZBMRHQCKA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.745717487 (604h4m51.616800522s) [noise=true , address=64.49.13.170:4717 , piece_id=NSYCFF26VLJ6QPF5JAPBZRBLULLDAWUEPHI5HZTPSXQVX43KQHVQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.782268034 (604h4m51.653351069s) [noise=true , address=64.49.13.138:4483 , piece_id=GRAGOWDG4I5TDHVNX75QJBTS7NIIRY3B73VWDO7DWNITNJXBAYWQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.856613327 (604h4m51.727696362s) [noise=true , address=145.79.227.52:4458 , piece_id=2XCSF7MRBBZBGEJ2GE63K25BO5DFB7DUSZ4KXMFRS5NOOHPSEJRA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.900758014 (604h4m51.771841059s) [noise=true , address=77.90.31.10:4534 , piece_id=KPMW47ZBIINLCZ67FH6EGEDFLEAT7F35CYP62D4TT7IBLQ57I2FA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.968806577 (604h4m51.839889622s) [noise=true , address=77.90.31.13:4631 , piece_id=IDYOBFP732AUWTYULX2I5ETBTS3DFAVW2EVEZ4BB5NYZFA4P2FTA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.000820185 (604h4m51.87190323s) [noise=true , address=45.91.116.14:4199 , piece_id=J3LDA3N2Q6B3VTWS5LG25LUD5E2L7I3FD56JPOVJVCJYLXC3AKXQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.111919172 (604h4m51.983002207s) [noise=true , address=145.79.227.34:4908 , piece_id=X57UEZNRHXKLR7R3VEWRKCALEHCMTE7NR73QGALOEZ7NKOBKDDFA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.513345511 (604h4m52.384428556s) [noise=true , address=5.231.73.4:4027 , piece_id=DVNFM76WSUGT65ZKJY4RFVXXLZIDQJ7TQ5CYBPUMZ2GXQWJOIY3Q, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.525239208 (604h4m52.396322253s) [noise=true , address=145.79.227.49:4362 , piece_id=UXLOT5FU522U73SMQEUIZ4K43WTQEAV5XRVSGG6EWQON43UWQAYQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.547688835 (604h4m52.41877187s) [noise=true , address=145.79.227.30:4793 , piece_id=37HWPUMM2Q4CZDYKXZIKL5KKPEH4O6ASHI675HEP2K7EOCGBVLFA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.777085496 (604h4m52.648168531s) [noise=true , address=145.79.227.33:4887 , piece_id=I7CHA6ZOBOS7J4SHID6MQLGAVR66BZ2D2J4VSTFCQW525BPN5KHQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.777110584 (604h4m52.648193619s) [seg_pos={0 0}, upload=529467]: Segment reached optimal threshold of 14 pieces. Dec 30 19:33:52.777320053 (604h4m52.648403088s) [noise=true , address=45.91.116.9:4288 , piece_id=QLRNLFOW2RZ6JLJDNJEKBQ6JWJLOZ57NBPADF3KYVLL4L5FTC62A, seg_pos={0 0}, upload=529467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX): protocol: write tcp 188.166.225.170:38490->45.91.116.9:4288: use of closed network connection; write tcp 188.166.225.170:38490->45.91.116.9:4288: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38490->45.91.116.9:4288: use of closed network connection Dec 30 19:33:52.777326334 (604h4m52.64840938s) [noise=true , address=45.91.116.9:4288 , piece_id=QLRNLFOW2RZ6JLJDNJEKBQ6JWJLOZ57NBPADF3KYVLL4L5FTC62A, seg_pos={0 0}, upload=529467]: Long tail cancellation due to optimal threshold Dec 30 19:33:52.777615194 (604h4m52.648698229s) [noise=true , address=77.90.31.14:4669 , piece_id=O2Z45BXHUPJQZZ7EJG4FKS3MNDPHWPBT4EMZCZV2HTJSEVAI7WHQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T): protocol: expected piece hash; context canceled; EOF Dec 30 19:33:52.777622820 (604h4m52.648705855s) [noise=true , address=77.90.31.14:4669 , piece_id=O2Z45BXHUPJQZZ7EJG4FKS3MNDPHWPBT4EMZCZV2HTJSEVAI7WHQ, seg_pos={0 0}, upload=529467]: Long tail cancellation due to optimal threshold Dec 30 19:33:52.777648788 (604h4m52.648731823s) [noise=true , address=64.49.13.169:4708 , piece_id=BXCIK6V4R5LZHWUTJOCL66442ZM6T7IYWD2OH5OM2SM7GX5JBIZA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:36148->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:36148->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36148->64.49.13.169:4708: use of closed network connection Dec 30 19:33:52.777665260 (604h4m52.648748295s) [noise=true , address=64.49.13.169:4708 , piece_id=BXCIK6V4R5LZHWUTJOCL66442ZM6T7IYWD2OH5OM2SM7GX5JBIZA, seg_pos={0 0}, upload=529467]: Long tail cancellation due to optimal threshold Dec 30 19:33:52.777872405 (604h4m52.64895545s) [noise=true , address=145.79.227.4:4218 , piece_id=2H4DI7772BR4JLLGKJYTSKTKVOABDB4T6UJ74OBTLXHV23WPN5HA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mbzxg1BjmbXeHxGgheZmQoN91k8NgaAUUASPkJe3fEmS4eEkc): protocol: expected piece hash; context canceled; EOF Dec 30 19:33:52.777879939 (604h4m52.648962984s) [noise=true , address=145.79.227.4:4218 , piece_id=2H4DI7772BR4JLLGKJYTSKTKVOABDB4T6UJ74OBTLXHV23WPN5HA, seg_pos={0 0}, upload=529467]: Long tail cancellation due to optimal threshold Dec 30 19:33:52.777898815 (604h4m52.64898185s) [seg_pos={0 0}, upload=529467]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 19:33:52.777974709 (604h4m52.649057755s) [upload=529467]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:33:52.777979669 (604h4m52.649062714s) [upload=529467]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:33:53.082585408 (604h4m52.953668453s) [upload=529467]: Done uploading segments... Dec 30 19:33:53.082605185 (604h4m52.953688221s) [upload=529467]: Upload finished. err: <nil> Duration: 37.951021098s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X..X..X.....X..XX...X..X.....XX.X.....X.....X...............] duration: 7.798423969s trace: https://jaeger.dev.storj.io/trace/4b57a4d55c27f17d FAIL (2025-12-30 19:48:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 2MXNCFG5BZSBRLTW4OPJXJWAVDPMHK4GXYPWT6GBFIH65F3IEMVA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: UU2AYD5D3SDGOW7JMKI6CV2SNPTVZNKJUW4TFNWSQ22WH4LAOYTQ) context deadline exceeded ======LOGS====== Dec 30 19:37:18.017831880 (604h8m17.888914925s) [upload=529518]: Starting upload Dec 30 19:37:18.017920869 (604h8m17.889003914s) [upload=529518]: Done starting upload Dec 30 19:37:18.017989800 (604h8m17.889072835s) [upload=529518]: Uploading segments... Dec 30 19:37:18.018167178 (604h8m17.889250213s) [upload=529518]: Got next segment. Inline: false Dec 30 19:37:18.018211322 (604h8m17.889294367s) [upload=529518]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 19:37:18.018216993 (604h8m17.889300028s) [upload=529518]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 19:37:18.265194540 (604h8m18.136277575s) [seg_pos={0 0}, upload=529518]: Begin upload segment... Dec 30 19:37:18.265242221 (604h8m18.136325267s) [seg_pos={0 0}, upload=529518]: Done begin upload segment. Dec 30 19:37:18.265271887 (604h8m18.136354922s) [noise=true , address=45.91.116.13:4156 , piece_id=ZXMVGKUW2Q4SQNBGD5ZZP7CHF2RFVVLSZIYEMVOOZB2VU4CEU6UQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.265280745 (604h8m18.13636378s) [noise=true , address=145.79.227.22:4510 , piece_id=TFAYL6XXPFSLYYPUK5GRBO7FDDDAJRMPGI2PTD3SH7OLW6XR5QGQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.265996978 (604h8m18.137080013s) [noise=true , address=145.79.227.13:4803 , piece_id=DGVTZAR4T33Z4ZWIRBWKLPTY45W7CDF5SPJ7IOSUBFLNQWAD4XOQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.266478024 (604h8m18.137561059s) [noise=true , address=45.91.116.8:4048 , piece_id=2MXNCFG5BZSBRLTW4OPJXJWAVDPMHK4GXYPWT6GBFIH65F3IEMVA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.267063399 (604h8m18.138146444s) [noise=true , address=145.79.227.12:4765 , piece_id=D4M33JNPTNF5EOTP6V4CH2KBZOJBKIRZ5RCEUMJMFSB5KEO3EITA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.267602044 (604h8m18.138685079s) [noise=true , address=45.91.116.11:4118 , piece_id=UU2AYD5D3SDGOW7JMKI6CV2SNPTVZNKJUW4TFNWSQ22WH4LAOYTQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.268103920 (604h8m18.139186965s) [noise=true , address=145.79.227.24:4582 , piece_id=DWQXXGUEQ4IXJJXA57AIULGBQTCM5LMRS6IKSDKNVDXIBAS3HMRA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.268624721 (604h8m18.139707756s) [noise=true , address=145.79.227.2:4114 , piece_id=5ELY6KB4QBNBOPMG6ALNYRCW3CYXRYVSDGMRAJYYYPE6TZSMA4OA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.269149150 (604h8m18.140232195s) [noise=true , address=45.91.116.7:4012 , piece_id=LOW2MEEFBV64QEH7MKQT2R4PHUBPWPRZGDYVCYMELUD4BOQTGEVA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.269680512 (604h8m18.140763547s) [noise=true , address=77.90.31.12:4589 , piece_id=RQH5BFFWHPXUA4NMBZUJYNOAI5OQU43HDCGKJMYBSS6O3YDFQL2A, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.270313487 (604h8m18.141396522s) [noise=true , address=145.79.227.46:4601 , piece_id=Y4BKLMSLP2SBMSKC3S6ISJCIRAXTZMIUXQDTRYXBRIEE6V4YQYLA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.270669034 (604h8m18.141752069s) [noise=true , address=45.91.116.12:4227 , piece_id=Q26IH3AZJJISAEL5LPYT7CRFTP4KUMXOX44MYSG3BCLZNLMYLERQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.270947995 (604h8m18.14203103s) [noise=true , address=145.79.227.23:4547 , piece_id=6G7N6QNXE3HHSWO6BMDRCQS26RJKV7PETS3VCAPLPCJNFHR4WGEA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.271235693 (604h8m18.142318738s) [noise=true , address=77.90.31.21:4880 , piece_id=2EIJYMPCZP4BWQX4I3RSQS5IDOEAMX7UAGQIXEVRJQG6MPBZSFQA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.271512129 (604h8m18.142595174s) [noise=true , address=145.79.227.51:4410 , piece_id=65NLCMFGMV2NLVDAYRXQMYVQDBAWQT4TUAFRIYDUZOGT46S5KYGA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.271809736 (604h8m18.142892782s) [noise=true , address=145.79.227.29:4941 , piece_id=J2QTCA233FRN5E6KQXXIP767JHUL46VQJ7I76UTJEWVLO4UYAHFQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:19.968123940 (604h8m19.839206985s) [upload=529518]: Next returned nil segment Dec 30 19:37:19.968138839 (604h8m19.839221874s) [upload=529518]: Waiting for error group managing segments... Dec 30 19:37:20.240602244 (604h8m20.111685279s) [noise=true , address=145.79.227.13:4803 , piece_id=DGVTZAR4T33Z4ZWIRBWKLPTY45W7CDF5SPJ7IOSUBFLNQWAD4XOQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.242668559 (604h8m20.113751594s) [noise=true , address=145.79.227.2:4114 , piece_id=5ELY6KB4QBNBOPMG6ALNYRCW3CYXRYVSDGMRAJYYYPE6TZSMA4OA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.274323484 (604h8m20.145406529s) [noise=true , address=145.79.227.51:4410 , piece_id=65NLCMFGMV2NLVDAYRXQMYVQDBAWQT4TUAFRIYDUZOGT46S5KYGA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.276151625 (604h8m20.14723467s) [noise=true , address=145.79.227.24:4582 , piece_id=DWQXXGUEQ4IXJJXA57AIULGBQTCM5LMRS6IKSDKNVDXIBAS3HMRA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.325870757 (604h8m20.196953792s) [noise=true , address=145.79.227.46:4601 , piece_id=Y4BKLMSLP2SBMSKC3S6ISJCIRAXTZMIUXQDTRYXBRIEE6V4YQYLA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.338603552 (604h8m20.209686587s) [noise=true , address=145.79.227.22:4510 , piece_id=TFAYL6XXPFSLYYPUK5GRBO7FDDDAJRMPGI2PTD3SH7OLW6XR5QGQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.339426289 (604h8m20.210509334s) [noise=true , address=145.79.227.12:4765 , piece_id=D4M33JNPTNF5EOTP6V4CH2KBZOJBKIRZ5RCEUMJMFSB5KEO3EITA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.528772708 (604h8m20.399855963s) [noise=true , address=77.90.31.21:4880 , piece_id=2EIJYMPCZP4BWQX4I3RSQS5IDOEAMX7UAGQIXEVRJQG6MPBZSFQA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.723971783 (604h8m20.595054818s) [noise=true , address=45.91.116.12:4227 , piece_id=Q26IH3AZJJISAEL5LPYT7CRFTP4KUMXOX44MYSG3BCLZNLMYLERQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.738010033 (604h8m20.609093068s) [noise=true , address=45.91.116.7:4012 , piece_id=LOW2MEEFBV64QEH7MKQT2R4PHUBPWPRZGDYVCYMELUD4BOQTGEVA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.900656063 (604h8m20.771739098s) [noise=true , address=45.91.116.8:4048 , piece_id=2MXNCFG5BZSBRLTW4OPJXJWAVDPMHK4GXYPWT6GBFIH65F3IEMVA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:21.200841094 (604h8m21.07192414s) [noise=true , address=77.90.31.12:4589 , piece_id=RQH5BFFWHPXUA4NMBZUJYNOAI5OQU43HDCGKJMYBSS6O3YDFQL2A, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:21.316516896 (604h8m21.187599941s) [noise=true , address=45.91.116.11:4118 , piece_id=UU2AYD5D3SDGOW7JMKI6CV2SNPTVZNKJUW4TFNWSQ22WH4LAOYTQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:21.316538046 (604h8m21.187621081s) [seg_pos={0 0}, upload=529518]: Segment reached optimal threshold of 13 pieces. Dec 30 19:37:21.316716626 (604h8m21.187799661s) [noise=true , address=145.79.227.29:4941 , piece_id=J2QTCA233FRN5E6KQXXIP767JHUL46VQJ7I76UTJEWVLO4UYAHFQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 30 19:37:21.316723229 (604h8m21.187806264s) [noise=true , address=145.79.227.29:4941 , piece_id=J2QTCA233FRN5E6KQXXIP767JHUL46VQJ7I76UTJEWVLO4UYAHFQ, seg_pos={0 0}, upload=529518]: Long tail cancellation due to optimal threshold Dec 30 19:37:21.316823179 (604h8m21.187906214s) [noise=true , address=45.91.116.13:4156 , piece_id=ZXMVGKUW2Q4SQNBGD5ZZP7CHF2RFVVLSZIYEMVOOZB2VU4CEU6UQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 30 19:37:21.316831525 (604h8m21.18791456s) [noise=true , address=45.91.116.13:4156 , piece_id=ZXMVGKUW2Q4SQNBGD5ZZP7CHF2RFVVLSZIYEMVOOZB2VU4CEU6UQ, seg_pos={0 0}, upload=529518]: Long tail cancellation due to optimal threshold Dec 30 19:37:21.317004414 (604h8m21.18808745s) [noise=true , address=145.79.227.23:4547 , piece_id=6G7N6QNXE3HHSWO6BMDRCQS26RJKV7PETS3VCAPLPCJNFHR4WGEA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: write tcp 188.166.225.170:45998->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:45998->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45998->145.79.227.23:4547: use of closed network connection Dec 30 19:37:21.317013822 (604h8m21.188096877s) [noise=true , address=145.79.227.23:4547 , piece_id=6G7N6QNXE3HHSWO6BMDRCQS26RJKV7PETS3VCAPLPCJNFHR4WGEA, seg_pos={0 0}, upload=529518]: Long tail cancellation due to optimal threshold Dec 30 19:37:21.317033068 (604h8m21.188116113s) [seg_pos={0 0}, upload=529518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 19:37:21.317121227 (604h8m21.188204262s) [upload=529518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:37:21.317128490 (604h8m21.188211555s) [upload=529518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:37:21.616086595 (604h8m21.48716965s) [upload=529518]: Done uploading segments... Dec 30 19:37:21.616243293 (604h8m21.487326328s) [upload=529518]: Upload finished. err: <nil> Duration: 7.798449477s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X..X.....................X..................] duration: 15.430521858s trace: https://jaeger.dev.storj.io/trace/2e4bd3c87074131f FAIL (2025-12-30 19:48:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== 98432889s) [noise=true , address=145.79.227.49:4362 , piece_id=UXLOT5FU522U73SMQEUIZ4K43WTQEAV5XRVSGG6EWQON43UWQAYQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.827625579 (604h4m49.698708614s) [noise=true , address=145.79.227.4:4218 , piece_id=2H4DI7772BR4JLLGKJYTSKTKVOABDB4T6UJ74OBTLXHV23WPN5HA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.827942963 (604h4m49.699025998s) [noise=true , address=45.91.116.9:4288 , piece_id=QLRNLFOW2RZ6JLJDNJEKBQ6JWJLOZ57NBPADF3KYVLL4L5FTC62A, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.828225852 (604h4m49.699308887s) [noise=true , address=45.91.116.14:4199 , piece_id=J3LDA3N2Q6B3VTWS5LG25LUD5E2L7I3FD56JPOVJVCJYLXC3AKXQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.828502729 (604h4m49.699585764s) [noise=true , address=64.49.13.138:4483 , piece_id=GRAGOWDG4I5TDHVNX75QJBTS7NIIRY3B73VWDO7DWNITNJXBAYWQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.828558776 (604h4m49.699641821s) [noise=true , address=64.49.13.169:4708 , piece_id=BXCIK6V4R5LZHWUTJOCL66442ZM6T7IYWD2OH5OM2SM7GX5JBIZA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.828853557 (604h4m49.699936592s) [noise=true , address=145.79.227.50:4384 , piece_id=U5R4PT66AXFJFPAGFADCVX6NNNIFLFM3CFTREKVEOOUDHTVTUTBA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.829160732 (604h4m49.700243767s) [noise=true , address=5.231.73.4:4027 , piece_id=DVNFM76WSUGT65ZKJY4RFVXXLZIDQJ7TQ5CYBPUMZ2GXQWJOIY3Q, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.829534374 (604h4m49.700617409s) [noise=true , address=145.79.227.52:4458 , piece_id=2XCSF7MRBBZBGEJ2GE63K25BO5DFB7DUSZ4KXMFRS5NOOHPSEJRA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.829873299 (604h4m49.700956334s) [noise=true , address=145.79.227.34:4908 , piece_id=X57UEZNRHXKLR7R3VEWRKCALEHCMTE7NR73QGALOEZ7NKOBKDDFA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.830200823 (604h4m49.701283868s) [noise=true , address=145.79.227.33:4887 , piece_id=I7CHA6ZOBOS7J4SHID6MQLGAVR66BZ2D2J4VSTFCQW525BPN5KHQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.838625865 (604h4m49.7097089s) [noise=true , address=145.79.227.3:4180 , piece_id=CYOSWRJR37YSHNJBWZZPMOLM7TVRJ2GFEV4XNNHRAQJWFU6Y2JIQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.839781686 (604h4m49.710864731s) [noise=true , address=145.79.227.36:4546 , piece_id=4V7GWX3MULB6CR3QH44ZIOISDONXKJ334XFSYHXUECJZBMRHQCKA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.840846864 (604h4m49.711929899s) [noise=true , address=77.90.31.14:4669 , piece_id=O2Z45BXHUPJQZZ7EJG4FKS3MNDPHWPBT4EMZCZV2HTJSEVAI7WHQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.841958571 (604h4m49.713041606s) [noise=true , address=77.90.31.10:4534 , piece_id=KPMW47ZBIINLCZ67FH6EGEDFLEAT7F35CYP62D4TT7IBLQ57I2FA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:51.267701872 (604h4m51.138784917s) [upload=529467]: Next returned nil segment Dec 30 19:33:51.267719867 (604h4m51.138802912s) [upload=529467]: Waiting for error group managing segments... Dec 30 19:33:51.700925266 (604h4m51.572008302s) [noise=true , address=145.79.227.3:4180 , piece_id=CYOSWRJR37YSHNJBWZZPMOLM7TVRJ2GFEV4XNNHRAQJWFU6Y2JIQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.707415294 (604h4m51.578498329s) [noise=true , address=145.79.227.50:4384 , piece_id=U5R4PT66AXFJFPAGFADCVX6NNNIFLFM3CFTREKVEOOUDHTVTUTBA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.732565163 (604h4m51.603648198s) [noise=true , address=145.79.227.36:4546 , piece_id=4V7GWX3MULB6CR3QH44ZIOISDONXKJ334XFSYHXUECJZBMRHQCKA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.745717487 (604h4m51.616800522s) [noise=true , address=64.49.13.170:4717 , piece_id=NSYCFF26VLJ6QPF5JAPBZRBLULLDAWUEPHI5HZTPSXQVX43KQHVQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.782268034 (604h4m51.653351069s) [noise=true , address=64.49.13.138:4483 , piece_id=GRAGOWDG4I5TDHVNX75QJBTS7NIIRY3B73VWDO7DWNITNJXBAYWQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.856613327 (604h4m51.727696362s) [noise=true , address=145.79.227.52:4458 , piece_id=2XCSF7MRBBZBGEJ2GE63K25BO5DFB7DUSZ4KXMFRS5NOOHPSEJRA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.900758014 (604h4m51.771841059s) [noise=true , address=77.90.31.10:4534 , piece_id=KPMW47ZBIINLCZ67FH6EGEDFLEAT7F35CYP62D4TT7IBLQ57I2FA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.968806577 (604h4m51.839889622s) [noise=true , address=77.90.31.13:4631 , piece_id=IDYOBFP732AUWTYULX2I5ETBTS3DFAVW2EVEZ4BB5NYZFA4P2FTA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.000820185 (604h4m51.87190323s) [noise=true , address=45.91.116.14:4199 , piece_id=J3LDA3N2Q6B3VTWS5LG25LUD5E2L7I3FD56JPOVJVCJYLXC3AKXQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.111919172 (604h4m51.983002207s) [noise=true , address=145.79.227.34:4908 , piece_id=X57UEZNRHXKLR7R3VEWRKCALEHCMTE7NR73QGALOEZ7NKOBKDDFA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.513345511 (604h4m52.384428556s) [noise=true , address=5.231.73.4:4027 , piece_id=DVNFM76WSUGT65ZKJY4RFVXXLZIDQJ7TQ5CYBPUMZ2GXQWJOIY3Q, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.525239208 (604h4m52.396322253s) [noise=true , address=145.79.227.49:4362 , piece_id=UXLOT5FU522U73SMQEUIZ4K43WTQEAV5XRVSGG6EWQON43UWQAYQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.547688835 (604h4m52.41877187s) [noise=true , address=145.79.227.30:4793 , piece_id=37HWPUMM2Q4CZDYKXZIKL5KKPEH4O6ASHI675HEP2K7EOCGBVLFA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.777085496 (604h4m52.648168531s) [noise=true , address=145.79.227.33:4887 , piece_id=I7CHA6ZOBOS7J4SHID6MQLGAVR66BZ2D2J4VSTFCQW525BPN5KHQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.777110584 (604h4m52.648193619s) [seg_pos={0 0}, upload=529467]: Segment reached optimal threshold of 14 pieces. Dec 30 19:33:52.777320053 (604h4m52.648403088s) [noise=true , address=45.91.116.9:4288 , piece_id=QLRNLFOW2RZ6JLJDNJEKBQ6JWJLOZ57NBPADF3KYVLL4L5FTC62A, seg_pos={0 0}, upload=529467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX): protocol: write tcp 188.166.225.170:38490->45.91.116.9:4288: use of closed network connection; write tcp 188.166.225.170:38490->45.91.116.9:4288: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38490->45.91.116.9:4288: use of closed network connection Dec 30 19:33:52.777326334 (604h4m52.64840938s) [noise=true , address=45.91.116.9:4288 , piece_id=QLRNLFOW2RZ6JLJDNJEKBQ6JWJLOZ57NBPADF3KYVLL4L5FTC62A, seg_pos={0 0}, upload=529467]: Long tail cancellation due to optimal threshold Dec 30 19:33:52.777615194 (604h4m52.648698229s) [noise=true , address=77.90.31.14:4669 , piece_id=O2Z45BXHUPJQZZ7EJG4FKS3MNDPHWPBT4EMZCZV2HTJSEVAI7WHQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T): protocol: expected piece hash; context canceled; EOF Dec 30 19:33:52.777622820 (604h4m52.648705855s) [noise=true , address=77.90.31.14:4669 , piece_id=O2Z45BXHUPJQZZ7EJG4FKS3MNDPHWPBT4EMZCZV2HTJSEVAI7WHQ, seg_pos={0 0}, upload=529467]: Long tail cancellation due to optimal threshold Dec 30 19:33:52.777648788 (604h4m52.648731823s) [noise=true , address=64.49.13.169:4708 , piece_id=BXCIK6V4R5LZHWUTJOCL66442ZM6T7IYWD2OH5OM2SM7GX5JBIZA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:36148->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:36148->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36148->64.49.13.169:4708: use of closed network connection Dec 30 19:33:52.777665260 (604h4m52.648748295s) [noise=true , address=64.49.13.169:4708 , piece_id=BXCIK6V4R5LZHWUTJOCL66442ZM6T7IYWD2OH5OM2SM7GX5JBIZA, seg_pos={0 0}, upload=529467]: Long tail cancellation due to optimal threshold Dec 30 19:33:52.777872405 (604h4m52.64895545s) [noise=true , address=145.79.227.4:4218 , piece_id=2H4DI7772BR4JLLGKJYTSKTKVOABDB4T6UJ74OBTLXHV23WPN5HA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mbzxg1BjmbXeHxGgheZmQoN91k8NgaAUUASPkJe3fEmS4eEkc): protocol: expected piece hash; context canceled; EOF Dec 30 19:33:52.777879939 (604h4m52.648962984s) [noise=true , address=145.79.227.4:4218 , piece_id=2H4DI7772BR4JLLGKJYTSKTKVOABDB4T6UJ74OBTLXHV23WPN5HA, seg_pos={0 0}, upload=529467]: Long tail cancellation due to optimal threshold Dec 30 19:33:52.777898815 (604h4m52.64898185s) [seg_pos={0 0}, upload=529467]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 19:33:52.777974709 (604h4m52.649057755s) [upload=529467]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:33:52.777979669 (604h4m52.649062714s) [upload=529467]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:33:53.082585408 (604h4m52.953668453s) [upload=529467]: Done uploading segments... Dec 30 19:33:53.082605185 (604h4m52.953688221s) [upload=529467]: Upload finished. err: <nil> Duration: 15.430545923s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X..X..X.....X..XX...X..X.....XX.X.....X.....X..............] duration: 5.816630149s trace: https://jaeger.dev.storj.io/trace/4b57a4d55c27f17d FAIL (2025-12-30 19:47:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 2MXNCFG5BZSBRLTW4OPJXJWAVDPMHK4GXYPWT6GBFIH65F3IEMVA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: UU2AYD5D3SDGOW7JMKI6CV2SNPTVZNKJUW4TFNWSQ22WH4LAOYTQ) context deadline exceeded ======LOGS====== Dec 30 19:37:18.017831880 (604h8m17.888914925s) [upload=529518]: Starting upload Dec 30 19:37:18.017920869 (604h8m17.889003914s) [upload=529518]: Done starting upload Dec 30 19:37:18.017989800 (604h8m17.889072835s) [upload=529518]: Uploading segments... Dec 30 19:37:18.018167178 (604h8m17.889250213s) [upload=529518]: Got next segment. Inline: false Dec 30 19:37:18.018211322 (604h8m17.889294367s) [upload=529518]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 19:37:18.018216993 (604h8m17.889300028s) [upload=529518]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 19:37:18.265194540 (604h8m18.136277575s) [seg_pos={0 0}, upload=529518]: Begin upload segment... Dec 30 19:37:18.265242221 (604h8m18.136325267s) [seg_pos={0 0}, upload=529518]: Done begin upload segment. Dec 30 19:37:18.265271887 (604h8m18.136354922s) [noise=true , address=45.91.116.13:4156 , piece_id=ZXMVGKUW2Q4SQNBGD5ZZP7CHF2RFVVLSZIYEMVOOZB2VU4CEU6UQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.265280745 (604h8m18.13636378s) [noise=true , address=145.79.227.22:4510 , piece_id=TFAYL6XXPFSLYYPUK5GRBO7FDDDAJRMPGI2PTD3SH7OLW6XR5QGQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.265996978 (604h8m18.137080013s) [noise=true , address=145.79.227.13:4803 , piece_id=DGVTZAR4T33Z4ZWIRBWKLPTY45W7CDF5SPJ7IOSUBFLNQWAD4XOQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.266478024 (604h8m18.137561059s) [noise=true , address=45.91.116.8:4048 , piece_id=2MXNCFG5BZSBRLTW4OPJXJWAVDPMHK4GXYPWT6GBFIH65F3IEMVA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.267063399 (604h8m18.138146444s) [noise=true , address=145.79.227.12:4765 , piece_id=D4M33JNPTNF5EOTP6V4CH2KBZOJBKIRZ5RCEUMJMFSB5KEO3EITA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.267602044 (604h8m18.138685079s) [noise=true , address=45.91.116.11:4118 , piece_id=UU2AYD5D3SDGOW7JMKI6CV2SNPTVZNKJUW4TFNWSQ22WH4LAOYTQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.268103920 (604h8m18.139186965s) [noise=true , address=145.79.227.24:4582 , piece_id=DWQXXGUEQ4IXJJXA57AIULGBQTCM5LMRS6IKSDKNVDXIBAS3HMRA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.268624721 (604h8m18.139707756s) [noise=true , address=145.79.227.2:4114 , piece_id=5ELY6KB4QBNBOPMG6ALNYRCW3CYXRYVSDGMRAJYYYPE6TZSMA4OA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.269149150 (604h8m18.140232195s) [noise=true , address=45.91.116.7:4012 , piece_id=LOW2MEEFBV64QEH7MKQT2R4PHUBPWPRZGDYVCYMELUD4BOQTGEVA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.269680512 (604h8m18.140763547s) [noise=true , address=77.90.31.12:4589 , piece_id=RQH5BFFWHPXUA4NMBZUJYNOAI5OQU43HDCGKJMYBSS6O3YDFQL2A, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.270313487 (604h8m18.141396522s) [noise=true , address=145.79.227.46:4601 , piece_id=Y4BKLMSLP2SBMSKC3S6ISJCIRAXTZMIUXQDTRYXBRIEE6V4YQYLA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.270669034 (604h8m18.141752069s) [noise=true , address=45.91.116.12:4227 , piece_id=Q26IH3AZJJISAEL5LPYT7CRFTP4KUMXOX44MYSG3BCLZNLMYLERQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.270947995 (604h8m18.14203103s) [noise=true , address=145.79.227.23:4547 , piece_id=6G7N6QNXE3HHSWO6BMDRCQS26RJKV7PETS3VCAPLPCJNFHR4WGEA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.271235693 (604h8m18.142318738s) [noise=true , address=77.90.31.21:4880 , piece_id=2EIJYMPCZP4BWQX4I3RSQS5IDOEAMX7UAGQIXEVRJQG6MPBZSFQA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.271512129 (604h8m18.142595174s) [noise=true , address=145.79.227.51:4410 , piece_id=65NLCMFGMV2NLVDAYRXQMYVQDBAWQT4TUAFRIYDUZOGT46S5KYGA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.271809736 (604h8m18.142892782s) [noise=true , address=145.79.227.29:4941 , piece_id=J2QTCA233FRN5E6KQXXIP767JHUL46VQJ7I76UTJEWVLO4UYAHFQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:19.968123940 (604h8m19.839206985s) [upload=529518]: Next returned nil segment Dec 30 19:37:19.968138839 (604h8m19.839221874s) [upload=529518]: Waiting for error group managing segments... Dec 30 19:37:20.240602244 (604h8m20.111685279s) [noise=true , address=145.79.227.13:4803 , piece_id=DGVTZAR4T33Z4ZWIRBWKLPTY45W7CDF5SPJ7IOSUBFLNQWAD4XOQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.242668559 (604h8m20.113751594s) [noise=true , address=145.79.227.2:4114 , piece_id=5ELY6KB4QBNBOPMG6ALNYRCW3CYXRYVSDGMRAJYYYPE6TZSMA4OA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.274323484 (604h8m20.145406529s) [noise=true , address=145.79.227.51:4410 , piece_id=65NLCMFGMV2NLVDAYRXQMYVQDBAWQT4TUAFRIYDUZOGT46S5KYGA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.276151625 (604h8m20.14723467s) [noise=true , address=145.79.227.24:4582 , piece_id=DWQXXGUEQ4IXJJXA57AIULGBQTCM5LMRS6IKSDKNVDXIBAS3HMRA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.325870757 (604h8m20.196953792s) [noise=true , address=145.79.227.46:4601 , piece_id=Y4BKLMSLP2SBMSKC3S6ISJCIRAXTZMIUXQDTRYXBRIEE6V4YQYLA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.338603552 (604h8m20.209686587s) [noise=true , address=145.79.227.22:4510 , piece_id=TFAYL6XXPFSLYYPUK5GRBO7FDDDAJRMPGI2PTD3SH7OLW6XR5QGQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.339426289 (604h8m20.210509334s) [noise=true , address=145.79.227.12:4765 , piece_id=D4M33JNPTNF5EOTP6V4CH2KBZOJBKIRZ5RCEUMJMFSB5KEO3EITA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.528772708 (604h8m20.399855963s) [noise=true , address=77.90.31.21:4880 , piece_id=2EIJYMPCZP4BWQX4I3RSQS5IDOEAMX7UAGQIXEVRJQG6MPBZSFQA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.723971783 (604h8m20.595054818s) [noise=true , address=45.91.116.12:4227 , piece_id=Q26IH3AZJJISAEL5LPYT7CRFTP4KUMXOX44MYSG3BCLZNLMYLERQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.738010033 (604h8m20.609093068s) [noise=true , address=45.91.116.7:4012 , piece_id=LOW2MEEFBV64QEH7MKQT2R4PHUBPWPRZGDYVCYMELUD4BOQTGEVA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.900656063 (604h8m20.771739098s) [noise=true , address=45.91.116.8:4048 , piece_id=2MXNCFG5BZSBRLTW4OPJXJWAVDPMHK4GXYPWT6GBFIH65F3IEMVA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:21.200841094 (604h8m21.07192414s) [noise=true , address=77.90.31.12:4589 , piece_id=RQH5BFFWHPXUA4NMBZUJYNOAI5OQU43HDCGKJMYBSS6O3YDFQL2A, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:21.316516896 (604h8m21.187599941s) [noise=true , address=45.91.116.11:4118 , piece_id=UU2AYD5D3SDGOW7JMKI6CV2SNPTVZNKJUW4TFNWSQ22WH4LAOYTQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:21.316538046 (604h8m21.187621081s) [seg_pos={0 0}, upload=529518]: Segment reached optimal threshold of 13 pieces. Dec 30 19:37:21.316716626 (604h8m21.187799661s) [noise=true , address=145.79.227.29:4941 , piece_id=J2QTCA233FRN5E6KQXXIP767JHUL46VQJ7I76UTJEWVLO4UYAHFQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 30 19:37:21.316723229 (604h8m21.187806264s) [noise=true , address=145.79.227.29:4941 , piece_id=J2QTCA233FRN5E6KQXXIP767JHUL46VQJ7I76UTJEWVLO4UYAHFQ, seg_pos={0 0}, upload=529518]: Long tail cancellation due to optimal threshold Dec 30 19:37:21.316823179 (604h8m21.187906214s) [noise=true , address=45.91.116.13:4156 , piece_id=ZXMVGKUW2Q4SQNBGD5ZZP7CHF2RFVVLSZIYEMVOOZB2VU4CEU6UQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 30 19:37:21.316831525 (604h8m21.18791456s) [noise=true , address=45.91.116.13:4156 , piece_id=ZXMVGKUW2Q4SQNBGD5ZZP7CHF2RFVVLSZIYEMVOOZB2VU4CEU6UQ, seg_pos={0 0}, upload=529518]: Long tail cancellation due to optimal threshold Dec 30 19:37:21.317004414 (604h8m21.18808745s) [noise=true , address=145.79.227.23:4547 , piece_id=6G7N6QNXE3HHSWO6BMDRCQS26RJKV7PETS3VCAPLPCJNFHR4WGEA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: write tcp 188.166.225.170:45998->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:45998->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45998->145.79.227.23:4547: use of closed network connection Dec 30 19:37:21.317013822 (604h8m21.188096877s) [noise=true , address=145.79.227.23:4547 , piece_id=6G7N6QNXE3HHSWO6BMDRCQS26RJKV7PETS3VCAPLPCJNFHR4WGEA, seg_pos={0 0}, upload=529518]: Long tail cancellation due to optimal threshold Dec 30 19:37:21.317033068 (604h8m21.188116113s) [seg_pos={0 0}, upload=529518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 19:37:21.317121227 (604h8m21.188204262s) [upload=529518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:37:21.317128490 (604h8m21.188211555s) [upload=529518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:37:21.616086595 (604h8m21.48716965s) [upload=529518]: Done uploading segments... Dec 30 19:37:21.616243293 (604h8m21.487326328s) [upload=529518]: Upload finished. err: <nil> Duration: 5.816650508s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..X..X..X.....X..XX...X..X.....XX.X.....X.....X.............] duration: 6.315886312s trace: https://jaeger.dev.storj.io/trace/4b57a4d55c27f17d FAIL (2025-12-30 19:47:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 2MXNCFG5BZSBRLTW4OPJXJWAVDPMHK4GXYPWT6GBFIH65F3IEMVA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: UU2AYD5D3SDGOW7JMKI6CV2SNPTVZNKJUW4TFNWSQ22WH4LAOYTQ) context deadline exceeded ======LOGS====== Dec 30 19:37:18.017831880 (604h8m17.888914925s) [upload=529518]: Starting upload Dec 30 19:37:18.017920869 (604h8m17.889003914s) [upload=529518]: Done starting upload Dec 30 19:37:18.017989800 (604h8m17.889072835s) [upload=529518]: Uploading segments... Dec 30 19:37:18.018167178 (604h8m17.889250213s) [upload=529518]: Got next segment. Inline: false Dec 30 19:37:18.018211322 (604h8m17.889294367s) [upload=529518]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 19:37:18.018216993 (604h8m17.889300028s) [upload=529518]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 19:37:18.265194540 (604h8m18.136277575s) [seg_pos={0 0}, upload=529518]: Begin upload segment... Dec 30 19:37:18.265242221 (604h8m18.136325267s) [seg_pos={0 0}, upload=529518]: Done begin upload segment. Dec 30 19:37:18.265271887 (604h8m18.136354922s) [noise=true , address=45.91.116.13:4156 , piece_id=ZXMVGKUW2Q4SQNBGD5ZZP7CHF2RFVVLSZIYEMVOOZB2VU4CEU6UQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.265280745 (604h8m18.13636378s) [noise=true , address=145.79.227.22:4510 , piece_id=TFAYL6XXPFSLYYPUK5GRBO7FDDDAJRMPGI2PTD3SH7OLW6XR5QGQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.265996978 (604h8m18.137080013s) [noise=true , address=145.79.227.13:4803 , piece_id=DGVTZAR4T33Z4ZWIRBWKLPTY45W7CDF5SPJ7IOSUBFLNQWAD4XOQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.266478024 (604h8m18.137561059s) [noise=true , address=45.91.116.8:4048 , piece_id=2MXNCFG5BZSBRLTW4OPJXJWAVDPMHK4GXYPWT6GBFIH65F3IEMVA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.267063399 (604h8m18.138146444s) [noise=true , address=145.79.227.12:4765 , piece_id=D4M33JNPTNF5EOTP6V4CH2KBZOJBKIRZ5RCEUMJMFSB5KEO3EITA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.267602044 (604h8m18.138685079s) [noise=true , address=45.91.116.11:4118 , piece_id=UU2AYD5D3SDGOW7JMKI6CV2SNPTVZNKJUW4TFNWSQ22WH4LAOYTQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.268103920 (604h8m18.139186965s) [noise=true , address=145.79.227.24:4582 , piece_id=DWQXXGUEQ4IXJJXA57AIULGBQTCM5LMRS6IKSDKNVDXIBAS3HMRA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.268624721 (604h8m18.139707756s) [noise=true , address=145.79.227.2:4114 , piece_id=5ELY6KB4QBNBOPMG6ALNYRCW3CYXRYVSDGMRAJYYYPE6TZSMA4OA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.269149150 (604h8m18.140232195s) [noise=true , address=45.91.116.7:4012 , piece_id=LOW2MEEFBV64QEH7MKQT2R4PHUBPWPRZGDYVCYMELUD4BOQTGEVA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.269680512 (604h8m18.140763547s) [noise=true , address=77.90.31.12:4589 , piece_id=RQH5BFFWHPXUA4NMBZUJYNOAI5OQU43HDCGKJMYBSS6O3YDFQL2A, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.270313487 (604h8m18.141396522s) [noise=true , address=145.79.227.46:4601 , piece_id=Y4BKLMSLP2SBMSKC3S6ISJCIRAXTZMIUXQDTRYXBRIEE6V4YQYLA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.270669034 (604h8m18.141752069s) [noise=true , address=45.91.116.12:4227 , piece_id=Q26IH3AZJJISAEL5LPYT7CRFTP4KUMXOX44MYSG3BCLZNLMYLERQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.270947995 (604h8m18.14203103s) [noise=true , address=145.79.227.23:4547 , piece_id=6G7N6QNXE3HHSWO6BMDRCQS26RJKV7PETS3VCAPLPCJNFHR4WGEA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.271235693 (604h8m18.142318738s) [noise=true , address=77.90.31.21:4880 , piece_id=2EIJYMPCZP4BWQX4I3RSQS5IDOEAMX7UAGQIXEVRJQG6MPBZSFQA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.271512129 (604h8m18.142595174s) [noise=true , address=145.79.227.51:4410 , piece_id=65NLCMFGMV2NLVDAYRXQMYVQDBAWQT4TUAFRIYDUZOGT46S5KYGA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.271809736 (604h8m18.142892782s) [noise=true , address=145.79.227.29:4941 , piece_id=J2QTCA233FRN5E6KQXXIP767JHUL46VQJ7I76UTJEWVLO4UYAHFQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:19.968123940 (604h8m19.839206985s) [upload=529518]: Next returned nil segment Dec 30 19:37:19.968138839 (604h8m19.839221874s) [upload=529518]: Waiting for error group managing segments... Dec 30 19:37:20.240602244 (604h8m20.111685279s) [noise=true , address=145.79.227.13:4803 , piece_id=DGVTZAR4T33Z4ZWIRBWKLPTY45W7CDF5SPJ7IOSUBFLNQWAD4XOQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.242668559 (604h8m20.113751594s) [noise=true , address=145.79.227.2:4114 , piece_id=5ELY6KB4QBNBOPMG6ALNYRCW3CYXRYVSDGMRAJYYYPE6TZSMA4OA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.274323484 (604h8m20.145406529s) [noise=true , address=145.79.227.51:4410 , piece_id=65NLCMFGMV2NLVDAYRXQMYVQDBAWQT4TUAFRIYDUZOGT46S5KYGA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.276151625 (604h8m20.14723467s) [noise=true , address=145.79.227.24:4582 , piece_id=DWQXXGUEQ4IXJJXA57AIULGBQTCM5LMRS6IKSDKNVDXIBAS3HMRA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.325870757 (604h8m20.196953792s) [noise=true , address=145.79.227.46:4601 , piece_id=Y4BKLMSLP2SBMSKC3S6ISJCIRAXTZMIUXQDTRYXBRIEE6V4YQYLA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.338603552 (604h8m20.209686587s) [noise=true , address=145.79.227.22:4510 , piece_id=TFAYL6XXPFSLYYPUK5GRBO7FDDDAJRMPGI2PTD3SH7OLW6XR5QGQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.339426289 (604h8m20.210509334s) [noise=true , address=145.79.227.12:4765 , piece_id=D4M33JNPTNF5EOTP6V4CH2KBZOJBKIRZ5RCEUMJMFSB5KEO3EITA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.528772708 (604h8m20.399855963s) [noise=true , address=77.90.31.21:4880 , piece_id=2EIJYMPCZP4BWQX4I3RSQS5IDOEAMX7UAGQIXEVRJQG6MPBZSFQA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.723971783 (604h8m20.595054818s) [noise=true , address=45.91.116.12:4227 , piece_id=Q26IH3AZJJISAEL5LPYT7CRFTP4KUMXOX44MYSG3BCLZNLMYLERQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.738010033 (604h8m20.609093068s) [noise=true , address=45.91.116.7:4012 , piece_id=LOW2MEEFBV64QEH7MKQT2R4PHUBPWPRZGDYVCYMELUD4BOQTGEVA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.900656063 (604h8m20.771739098s) [noise=true , address=45.91.116.8:4048 , piece_id=2MXNCFG5BZSBRLTW4OPJXJWAVDPMHK4GXYPWT6GBFIH65F3IEMVA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:21.200841094 (604h8m21.07192414s) [noise=true , address=77.90.31.12:4589 , piece_id=RQH5BFFWHPXUA4NMBZUJYNOAI5OQU43HDCGKJMYBSS6O3YDFQL2A, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:21.316516896 (604h8m21.187599941s) [noise=true , address=45.91.116.11:4118 , piece_id=UU2AYD5D3SDGOW7JMKI6CV2SNPTVZNKJUW4TFNWSQ22WH4LAOYTQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:21.316538046 (604h8m21.187621081s) [seg_pos={0 0}, upload=529518]: Segment reached optimal threshold of 13 pieces. Dec 30 19:37:21.316716626 (604h8m21.187799661s) [noise=true , address=145.79.227.29:4941 , piece_id=J2QTCA233FRN5E6KQXXIP767JHUL46VQJ7I76UTJEWVLO4UYAHFQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 30 19:37:21.316723229 (604h8m21.187806264s) [noise=true , address=145.79.227.29:4941 , piece_id=J2QTCA233FRN5E6KQXXIP767JHUL46VQJ7I76UTJEWVLO4UYAHFQ, seg_pos={0 0}, upload=529518]: Long tail cancellation due to optimal threshold Dec 30 19:37:21.316823179 (604h8m21.187906214s) [noise=true , address=45.91.116.13:4156 , piece_id=ZXMVGKUW2Q4SQNBGD5ZZP7CHF2RFVVLSZIYEMVOOZB2VU4CEU6UQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 30 19:37:21.316831525 (604h8m21.18791456s) [noise=true , address=45.91.116.13:4156 , piece_id=ZXMVGKUW2Q4SQNBGD5ZZP7CHF2RFVVLSZIYEMVOOZB2VU4CEU6UQ, seg_pos={0 0}, upload=529518]: Long tail cancellation due to optimal threshold Dec 30 19:37:21.317004414 (604h8m21.18808745s) [noise=true , address=145.79.227.23:4547 , piece_id=6G7N6QNXE3HHSWO6BMDRCQS26RJKV7PETS3VCAPLPCJNFHR4WGEA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: write tcp 188.166.225.170:45998->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:45998->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45998->145.79.227.23:4547: use of closed network connection Dec 30 19:37:21.317013822 (604h8m21.188096877s) [noise=true , address=145.79.227.23:4547 , piece_id=6G7N6QNXE3HHSWO6BMDRCQS26RJKV7PETS3VCAPLPCJNFHR4WGEA, seg_pos={0 0}, upload=529518]: Long tail cancellation due to optimal threshold Dec 30 19:37:21.317033068 (604h8m21.188116113s) [seg_pos={0 0}, upload=529518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 19:37:21.317121227 (604h8m21.188204262s) [upload=529518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:37:21.317128490 (604h8m21.188211555s) [upload=529518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:37:21.616086595 (604h8m21.48716965s) [upload=529518]: Done uploading segments... Dec 30 19:37:21.616243293 (604h8m21.487326328s) [upload=529518]: Upload finished. err: <nil> Duration: 6.315912272s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X..X.....................X.................] duration: 28.00864123s trace: https://jaeger.dev.storj.io/trace/2e4bd3c87074131f FAIL (2025-12-30 19:47:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== 98432889s) [noise=true , address=145.79.227.49:4362 , piece_id=UXLOT5FU522U73SMQEUIZ4K43WTQEAV5XRVSGG6EWQON43UWQAYQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.827625579 (604h4m49.698708614s) [noise=true , address=145.79.227.4:4218 , piece_id=2H4DI7772BR4JLLGKJYTSKTKVOABDB4T6UJ74OBTLXHV23WPN5HA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.827942963 (604h4m49.699025998s) [noise=true , address=45.91.116.9:4288 , piece_id=QLRNLFOW2RZ6JLJDNJEKBQ6JWJLOZ57NBPADF3KYVLL4L5FTC62A, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.828225852 (604h4m49.699308887s) [noise=true , address=45.91.116.14:4199 , piece_id=J3LDA3N2Q6B3VTWS5LG25LUD5E2L7I3FD56JPOVJVCJYLXC3AKXQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.828502729 (604h4m49.699585764s) [noise=true , address=64.49.13.138:4483 , piece_id=GRAGOWDG4I5TDHVNX75QJBTS7NIIRY3B73VWDO7DWNITNJXBAYWQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.828558776 (604h4m49.699641821s) [noise=true , address=64.49.13.169:4708 , piece_id=BXCIK6V4R5LZHWUTJOCL66442ZM6T7IYWD2OH5OM2SM7GX5JBIZA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.828853557 (604h4m49.699936592s) [noise=true , address=145.79.227.50:4384 , piece_id=U5R4PT66AXFJFPAGFADCVX6NNNIFLFM3CFTREKVEOOUDHTVTUTBA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.829160732 (604h4m49.700243767s) [noise=true , address=5.231.73.4:4027 , piece_id=DVNFM76WSUGT65ZKJY4RFVXXLZIDQJ7TQ5CYBPUMZ2GXQWJOIY3Q, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.829534374 (604h4m49.700617409s) [noise=true , address=145.79.227.52:4458 , piece_id=2XCSF7MRBBZBGEJ2GE63K25BO5DFB7DUSZ4KXMFRS5NOOHPSEJRA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.829873299 (604h4m49.700956334s) [noise=true , address=145.79.227.34:4908 , piece_id=X57UEZNRHXKLR7R3VEWRKCALEHCMTE7NR73QGALOEZ7NKOBKDDFA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.830200823 (604h4m49.701283868s) [noise=true , address=145.79.227.33:4887 , piece_id=I7CHA6ZOBOS7J4SHID6MQLGAVR66BZ2D2J4VSTFCQW525BPN5KHQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.838625865 (604h4m49.7097089s) [noise=true , address=145.79.227.3:4180 , piece_id=CYOSWRJR37YSHNJBWZZPMOLM7TVRJ2GFEV4XNNHRAQJWFU6Y2JIQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.839781686 (604h4m49.710864731s) [noise=true , address=145.79.227.36:4546 , piece_id=4V7GWX3MULB6CR3QH44ZIOISDONXKJ334XFSYHXUECJZBMRHQCKA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.840846864 (604h4m49.711929899s) [noise=true , address=77.90.31.14:4669 , piece_id=O2Z45BXHUPJQZZ7EJG4FKS3MNDPHWPBT4EMZCZV2HTJSEVAI7WHQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.841958571 (604h4m49.713041606s) [noise=true , address=77.90.31.10:4534 , piece_id=KPMW47ZBIINLCZ67FH6EGEDFLEAT7F35CYP62D4TT7IBLQ57I2FA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:51.267701872 (604h4m51.138784917s) [upload=529467]: Next returned nil segment Dec 30 19:33:51.267719867 (604h4m51.138802912s) [upload=529467]: Waiting for error group managing segments... Dec 30 19:33:51.700925266 (604h4m51.572008302s) [noise=true , address=145.79.227.3:4180 , piece_id=CYOSWRJR37YSHNJBWZZPMOLM7TVRJ2GFEV4XNNHRAQJWFU6Y2JIQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.707415294 (604h4m51.578498329s) [noise=true , address=145.79.227.50:4384 , piece_id=U5R4PT66AXFJFPAGFADCVX6NNNIFLFM3CFTREKVEOOUDHTVTUTBA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.732565163 (604h4m51.603648198s) [noise=true , address=145.79.227.36:4546 , piece_id=4V7GWX3MULB6CR3QH44ZIOISDONXKJ334XFSYHXUECJZBMRHQCKA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.745717487 (604h4m51.616800522s) [noise=true , address=64.49.13.170:4717 , piece_id=NSYCFF26VLJ6QPF5JAPBZRBLULLDAWUEPHI5HZTPSXQVX43KQHVQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.782268034 (604h4m51.653351069s) [noise=true , address=64.49.13.138:4483 , piece_id=GRAGOWDG4I5TDHVNX75QJBTS7NIIRY3B73VWDO7DWNITNJXBAYWQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.856613327 (604h4m51.727696362s) [noise=true , address=145.79.227.52:4458 , piece_id=2XCSF7MRBBZBGEJ2GE63K25BO5DFB7DUSZ4KXMFRS5NOOHPSEJRA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.900758014 (604h4m51.771841059s) [noise=true , address=77.90.31.10:4534 , piece_id=KPMW47ZBIINLCZ67FH6EGEDFLEAT7F35CYP62D4TT7IBLQ57I2FA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.968806577 (604h4m51.839889622s) [noise=true , address=77.90.31.13:4631 , piece_id=IDYOBFP732AUWTYULX2I5ETBTS3DFAVW2EVEZ4BB5NYZFA4P2FTA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.000820185 (604h4m51.87190323s) [noise=true , address=45.91.116.14:4199 , piece_id=J3LDA3N2Q6B3VTWS5LG25LUD5E2L7I3FD56JPOVJVCJYLXC3AKXQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.111919172 (604h4m51.983002207s) [noise=true , address=145.79.227.34:4908 , piece_id=X57UEZNRHXKLR7R3VEWRKCALEHCMTE7NR73QGALOEZ7NKOBKDDFA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.513345511 (604h4m52.384428556s) [noise=true , address=5.231.73.4:4027 , piece_id=DVNFM76WSUGT65ZKJY4RFVXXLZIDQJ7TQ5CYBPUMZ2GXQWJOIY3Q, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.525239208 (604h4m52.396322253s) [noise=true , address=145.79.227.49:4362 , piece_id=UXLOT5FU522U73SMQEUIZ4K43WTQEAV5XRVSGG6EWQON43UWQAYQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.547688835 (604h4m52.41877187s) [noise=true , address=145.79.227.30:4793 , piece_id=37HWPUMM2Q4CZDYKXZIKL5KKPEH4O6ASHI675HEP2K7EOCGBVLFA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.777085496 (604h4m52.648168531s) [noise=true , address=145.79.227.33:4887 , piece_id=I7CHA6ZOBOS7J4SHID6MQLGAVR66BZ2D2J4VSTFCQW525BPN5KHQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.777110584 (604h4m52.648193619s) [seg_pos={0 0}, upload=529467]: Segment reached optimal threshold of 14 pieces. Dec 30 19:33:52.777320053 (604h4m52.648403088s) [noise=true , address=45.91.116.9:4288 , piece_id=QLRNLFOW2RZ6JLJDNJEKBQ6JWJLOZ57NBPADF3KYVLL4L5FTC62A, seg_pos={0 0}, upload=529467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX): protocol: write tcp 188.166.225.170:38490->45.91.116.9:4288: use of closed network connection; write tcp 188.166.225.170:38490->45.91.116.9:4288: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38490->45.91.116.9:4288: use of closed network connection Dec 30 19:33:52.777326334 (604h4m52.64840938s) [noise=true , address=45.91.116.9:4288 , piece_id=QLRNLFOW2RZ6JLJDNJEKBQ6JWJLOZ57NBPADF3KYVLL4L5FTC62A, seg_pos={0 0}, upload=529467]: Long tail cancellation due to optimal threshold Dec 30 19:33:52.777615194 (604h4m52.648698229s) [noise=true , address=77.90.31.14:4669 , piece_id=O2Z45BXHUPJQZZ7EJG4FKS3MNDPHWPBT4EMZCZV2HTJSEVAI7WHQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T): protocol: expected piece hash; context canceled; EOF Dec 30 19:33:52.777622820 (604h4m52.648705855s) [noise=true , address=77.90.31.14:4669 , piece_id=O2Z45BXHUPJQZZ7EJG4FKS3MNDPHWPBT4EMZCZV2HTJSEVAI7WHQ, seg_pos={0 0}, upload=529467]: Long tail cancellation due to optimal threshold Dec 30 19:33:52.777648788 (604h4m52.648731823s) [noise=true , address=64.49.13.169:4708 , piece_id=BXCIK6V4R5LZHWUTJOCL66442ZM6T7IYWD2OH5OM2SM7GX5JBIZA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:36148->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:36148->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36148->64.49.13.169:4708: use of closed network connection Dec 30 19:33:52.777665260 (604h4m52.648748295s) [noise=true , address=64.49.13.169:4708 , piece_id=BXCIK6V4R5LZHWUTJOCL66442ZM6T7IYWD2OH5OM2SM7GX5JBIZA, seg_pos={0 0}, upload=529467]: Long tail cancellation due to optimal threshold Dec 30 19:33:52.777872405 (604h4m52.64895545s) [noise=true , address=145.79.227.4:4218 , piece_id=2H4DI7772BR4JLLGKJYTSKTKVOABDB4T6UJ74OBTLXHV23WPN5HA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mbzxg1BjmbXeHxGgheZmQoN91k8NgaAUUASPkJe3fEmS4eEkc): protocol: expected piece hash; context canceled; EOF Dec 30 19:33:52.777879939 (604h4m52.648962984s) [noise=true , address=145.79.227.4:4218 , piece_id=2H4DI7772BR4JLLGKJYTSKTKVOABDB4T6UJ74OBTLXHV23WPN5HA, seg_pos={0 0}, upload=529467]: Long tail cancellation due to optimal threshold Dec 30 19:33:52.777898815 (604h4m52.64898185s) [seg_pos={0 0}, upload=529467]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 19:33:52.777974709 (604h4m52.649057755s) [upload=529467]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:33:52.777979669 (604h4m52.649062714s) [upload=529467]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:33:53.082585408 (604h4m52.953668453s) [upload=529467]: Done uploading segments... Dec 30 19:33:53.082605185 (604h4m52.953688221s) [upload=529467]: Upload finished. err: <nil> Duration: 28.008670326s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...X..X..X.....X..XX...X..X.....XX.X.....X.....X............] duration: 6.236612612s trace: https://jaeger.dev.storj.io/trace/4b57a4d55c27f17d FAIL (2025-12-30 19:46:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 2MXNCFG5BZSBRLTW4OPJXJWAVDPMHK4GXYPWT6GBFIH65F3IEMVA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: UU2AYD5D3SDGOW7JMKI6CV2SNPTVZNKJUW4TFNWSQ22WH4LAOYTQ) context deadline exceeded ======LOGS====== Dec 30 19:37:18.017831880 (604h8m17.888914925s) [upload=529518]: Starting upload Dec 30 19:37:18.017920869 (604h8m17.889003914s) [upload=529518]: Done starting upload Dec 30 19:37:18.017989800 (604h8m17.889072835s) [upload=529518]: Uploading segments... Dec 30 19:37:18.018167178 (604h8m17.889250213s) [upload=529518]: Got next segment. Inline: false Dec 30 19:37:18.018211322 (604h8m17.889294367s) [upload=529518]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 19:37:18.018216993 (604h8m17.889300028s) [upload=529518]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 19:37:18.265194540 (604h8m18.136277575s) [seg_pos={0 0}, upload=529518]: Begin upload segment... Dec 30 19:37:18.265242221 (604h8m18.136325267s) [seg_pos={0 0}, upload=529518]: Done begin upload segment. Dec 30 19:37:18.265271887 (604h8m18.136354922s) [noise=true , address=45.91.116.13:4156 , piece_id=ZXMVGKUW2Q4SQNBGD5ZZP7CHF2RFVVLSZIYEMVOOZB2VU4CEU6UQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.265280745 (604h8m18.13636378s) [noise=true , address=145.79.227.22:4510 , piece_id=TFAYL6XXPFSLYYPUK5GRBO7FDDDAJRMPGI2PTD3SH7OLW6XR5QGQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.265996978 (604h8m18.137080013s) [noise=true , address=145.79.227.13:4803 , piece_id=DGVTZAR4T33Z4ZWIRBWKLPTY45W7CDF5SPJ7IOSUBFLNQWAD4XOQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.266478024 (604h8m18.137561059s) [noise=true , address=45.91.116.8:4048 , piece_id=2MXNCFG5BZSBRLTW4OPJXJWAVDPMHK4GXYPWT6GBFIH65F3IEMVA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.267063399 (604h8m18.138146444s) [noise=true , address=145.79.227.12:4765 , piece_id=D4M33JNPTNF5EOTP6V4CH2KBZOJBKIRZ5RCEUMJMFSB5KEO3EITA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.267602044 (604h8m18.138685079s) [noise=true , address=45.91.116.11:4118 , piece_id=UU2AYD5D3SDGOW7JMKI6CV2SNPTVZNKJUW4TFNWSQ22WH4LAOYTQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.268103920 (604h8m18.139186965s) [noise=true , address=145.79.227.24:4582 , piece_id=DWQXXGUEQ4IXJJXA57AIULGBQTCM5LMRS6IKSDKNVDXIBAS3HMRA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.268624721 (604h8m18.139707756s) [noise=true , address=145.79.227.2:4114 , piece_id=5ELY6KB4QBNBOPMG6ALNYRCW3CYXRYVSDGMRAJYYYPE6TZSMA4OA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.269149150 (604h8m18.140232195s) [noise=true , address=45.91.116.7:4012 , piece_id=LOW2MEEFBV64QEH7MKQT2R4PHUBPWPRZGDYVCYMELUD4BOQTGEVA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.269680512 (604h8m18.140763547s) [noise=true , address=77.90.31.12:4589 , piece_id=RQH5BFFWHPXUA4NMBZUJYNOAI5OQU43HDCGKJMYBSS6O3YDFQL2A, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.270313487 (604h8m18.141396522s) [noise=true , address=145.79.227.46:4601 , piece_id=Y4BKLMSLP2SBMSKC3S6ISJCIRAXTZMIUXQDTRYXBRIEE6V4YQYLA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.270669034 (604h8m18.141752069s) [noise=true , address=45.91.116.12:4227 , piece_id=Q26IH3AZJJISAEL5LPYT7CRFTP4KUMXOX44MYSG3BCLZNLMYLERQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.270947995 (604h8m18.14203103s) [noise=true , address=145.79.227.23:4547 , piece_id=6G7N6QNXE3HHSWO6BMDRCQS26RJKV7PETS3VCAPLPCJNFHR4WGEA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.271235693 (604h8m18.142318738s) [noise=true , address=77.90.31.21:4880 , piece_id=2EIJYMPCZP4BWQX4I3RSQS5IDOEAMX7UAGQIXEVRJQG6MPBZSFQA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.271512129 (604h8m18.142595174s) [noise=true , address=145.79.227.51:4410 , piece_id=65NLCMFGMV2NLVDAYRXQMYVQDBAWQT4TUAFRIYDUZOGT46S5KYGA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.271809736 (604h8m18.142892782s) [noise=true , address=145.79.227.29:4941 , piece_id=J2QTCA233FRN5E6KQXXIP767JHUL46VQJ7I76UTJEWVLO4UYAHFQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:19.968123940 (604h8m19.839206985s) [upload=529518]: Next returned nil segment Dec 30 19:37:19.968138839 (604h8m19.839221874s) [upload=529518]: Waiting for error group managing segments... Dec 30 19:37:20.240602244 (604h8m20.111685279s) [noise=true , address=145.79.227.13:4803 , piece_id=DGVTZAR4T33Z4ZWIRBWKLPTY45W7CDF5SPJ7IOSUBFLNQWAD4XOQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.242668559 (604h8m20.113751594s) [noise=true , address=145.79.227.2:4114 , piece_id=5ELY6KB4QBNBOPMG6ALNYRCW3CYXRYVSDGMRAJYYYPE6TZSMA4OA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.274323484 (604h8m20.145406529s) [noise=true , address=145.79.227.51:4410 , piece_id=65NLCMFGMV2NLVDAYRXQMYVQDBAWQT4TUAFRIYDUZOGT46S5KYGA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.276151625 (604h8m20.14723467s) [noise=true , address=145.79.227.24:4582 , piece_id=DWQXXGUEQ4IXJJXA57AIULGBQTCM5LMRS6IKSDKNVDXIBAS3HMRA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.325870757 (604h8m20.196953792s) [noise=true , address=145.79.227.46:4601 , piece_id=Y4BKLMSLP2SBMSKC3S6ISJCIRAXTZMIUXQDTRYXBRIEE6V4YQYLA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.338603552 (604h8m20.209686587s) [noise=true , address=145.79.227.22:4510 , piece_id=TFAYL6XXPFSLYYPUK5GRBO7FDDDAJRMPGI2PTD3SH7OLW6XR5QGQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.339426289 (604h8m20.210509334s) [noise=true , address=145.79.227.12:4765 , piece_id=D4M33JNPTNF5EOTP6V4CH2KBZOJBKIRZ5RCEUMJMFSB5KEO3EITA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.528772708 (604h8m20.399855963s) [noise=true , address=77.90.31.21:4880 , piece_id=2EIJYMPCZP4BWQX4I3RSQS5IDOEAMX7UAGQIXEVRJQG6MPBZSFQA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.723971783 (604h8m20.595054818s) [noise=true , address=45.91.116.12:4227 , piece_id=Q26IH3AZJJISAEL5LPYT7CRFTP4KUMXOX44MYSG3BCLZNLMYLERQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.738010033 (604h8m20.609093068s) [noise=true , address=45.91.116.7:4012 , piece_id=LOW2MEEFBV64QEH7MKQT2R4PHUBPWPRZGDYVCYMELUD4BOQTGEVA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.900656063 (604h8m20.771739098s) [noise=true , address=45.91.116.8:4048 , piece_id=2MXNCFG5BZSBRLTW4OPJXJWAVDPMHK4GXYPWT6GBFIH65F3IEMVA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:21.200841094 (604h8m21.07192414s) [noise=true , address=77.90.31.12:4589 , piece_id=RQH5BFFWHPXUA4NMBZUJYNOAI5OQU43HDCGKJMYBSS6O3YDFQL2A, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:21.316516896 (604h8m21.187599941s) [noise=true , address=45.91.116.11:4118 , piece_id=UU2AYD5D3SDGOW7JMKI6CV2SNPTVZNKJUW4TFNWSQ22WH4LAOYTQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:21.316538046 (604h8m21.187621081s) [seg_pos={0 0}, upload=529518]: Segment reached optimal threshold of 13 pieces. Dec 30 19:37:21.316716626 (604h8m21.187799661s) [noise=true , address=145.79.227.29:4941 , piece_id=J2QTCA233FRN5E6KQXXIP767JHUL46VQJ7I76UTJEWVLO4UYAHFQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 30 19:37:21.316723229 (604h8m21.187806264s) [noise=true , address=145.79.227.29:4941 , piece_id=J2QTCA233FRN5E6KQXXIP767JHUL46VQJ7I76UTJEWVLO4UYAHFQ, seg_pos={0 0}, upload=529518]: Long tail cancellation due to optimal threshold Dec 30 19:37:21.316823179 (604h8m21.187906214s) [noise=true , address=45.91.116.13:4156 , piece_id=ZXMVGKUW2Q4SQNBGD5ZZP7CHF2RFVVLSZIYEMVOOZB2VU4CEU6UQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 30 19:37:21.316831525 (604h8m21.18791456s) [noise=true , address=45.91.116.13:4156 , piece_id=ZXMVGKUW2Q4SQNBGD5ZZP7CHF2RFVVLSZIYEMVOOZB2VU4CEU6UQ, seg_pos={0 0}, upload=529518]: Long tail cancellation due to optimal threshold Dec 30 19:37:21.317004414 (604h8m21.18808745s) [noise=true , address=145.79.227.23:4547 , piece_id=6G7N6QNXE3HHSWO6BMDRCQS26RJKV7PETS3VCAPLPCJNFHR4WGEA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: write tcp 188.166.225.170:45998->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:45998->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45998->145.79.227.23:4547: use of closed network connection Dec 30 19:37:21.317013822 (604h8m21.188096877s) [noise=true , address=145.79.227.23:4547 , piece_id=6G7N6QNXE3HHSWO6BMDRCQS26RJKV7PETS3VCAPLPCJNFHR4WGEA, seg_pos={0 0}, upload=529518]: Long tail cancellation due to optimal threshold Dec 30 19:37:21.317033068 (604h8m21.188116113s) [seg_pos={0 0}, upload=529518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 19:37:21.317121227 (604h8m21.188204262s) [upload=529518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:37:21.317128490 (604h8m21.188211555s) [upload=529518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:37:21.616086595 (604h8m21.48716965s) [upload=529518]: Done uploading segments... Dec 30 19:37:21.616243293 (604h8m21.487326328s) [upload=529518]: Upload finished. err: <nil> Duration: 6.236644673s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X..X.....................X................] duration: 26.691842812s trace: https://jaeger.dev.storj.io/trace/2e4bd3c87074131f FAIL (2025-12-30 19:45:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== 98432889s) [noise=true , address=145.79.227.49:4362 , piece_id=UXLOT5FU522U73SMQEUIZ4K43WTQEAV5XRVSGG6EWQON43UWQAYQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.827625579 (604h4m49.698708614s) [noise=true , address=145.79.227.4:4218 , piece_id=2H4DI7772BR4JLLGKJYTSKTKVOABDB4T6UJ74OBTLXHV23WPN5HA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.827942963 (604h4m49.699025998s) [noise=true , address=45.91.116.9:4288 , piece_id=QLRNLFOW2RZ6JLJDNJEKBQ6JWJLOZ57NBPADF3KYVLL4L5FTC62A, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.828225852 (604h4m49.699308887s) [noise=true , address=45.91.116.14:4199 , piece_id=J3LDA3N2Q6B3VTWS5LG25LUD5E2L7I3FD56JPOVJVCJYLXC3AKXQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.828502729 (604h4m49.699585764s) [noise=true , address=64.49.13.138:4483 , piece_id=GRAGOWDG4I5TDHVNX75QJBTS7NIIRY3B73VWDO7DWNITNJXBAYWQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.828558776 (604h4m49.699641821s) [noise=true , address=64.49.13.169:4708 , piece_id=BXCIK6V4R5LZHWUTJOCL66442ZM6T7IYWD2OH5OM2SM7GX5JBIZA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.828853557 (604h4m49.699936592s) [noise=true , address=145.79.227.50:4384 , piece_id=U5R4PT66AXFJFPAGFADCVX6NNNIFLFM3CFTREKVEOOUDHTVTUTBA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.829160732 (604h4m49.700243767s) [noise=true , address=5.231.73.4:4027 , piece_id=DVNFM76WSUGT65ZKJY4RFVXXLZIDQJ7TQ5CYBPUMZ2GXQWJOIY3Q, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.829534374 (604h4m49.700617409s) [noise=true , address=145.79.227.52:4458 , piece_id=2XCSF7MRBBZBGEJ2GE63K25BO5DFB7DUSZ4KXMFRS5NOOHPSEJRA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.829873299 (604h4m49.700956334s) [noise=true , address=145.79.227.34:4908 , piece_id=X57UEZNRHXKLR7R3VEWRKCALEHCMTE7NR73QGALOEZ7NKOBKDDFA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.830200823 (604h4m49.701283868s) [noise=true , address=145.79.227.33:4887 , piece_id=I7CHA6ZOBOS7J4SHID6MQLGAVR66BZ2D2J4VSTFCQW525BPN5KHQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.838625865 (604h4m49.7097089s) [noise=true , address=145.79.227.3:4180 , piece_id=CYOSWRJR37YSHNJBWZZPMOLM7TVRJ2GFEV4XNNHRAQJWFU6Y2JIQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.839781686 (604h4m49.710864731s) [noise=true , address=145.79.227.36:4546 , piece_id=4V7GWX3MULB6CR3QH44ZIOISDONXKJ334XFSYHXUECJZBMRHQCKA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.840846864 (604h4m49.711929899s) [noise=true , address=77.90.31.14:4669 , piece_id=O2Z45BXHUPJQZZ7EJG4FKS3MNDPHWPBT4EMZCZV2HTJSEVAI7WHQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.841958571 (604h4m49.713041606s) [noise=true , address=77.90.31.10:4534 , piece_id=KPMW47ZBIINLCZ67FH6EGEDFLEAT7F35CYP62D4TT7IBLQ57I2FA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:51.267701872 (604h4m51.138784917s) [upload=529467]: Next returned nil segment Dec 30 19:33:51.267719867 (604h4m51.138802912s) [upload=529467]: Waiting for error group managing segments... Dec 30 19:33:51.700925266 (604h4m51.572008302s) [noise=true , address=145.79.227.3:4180 , piece_id=CYOSWRJR37YSHNJBWZZPMOLM7TVRJ2GFEV4XNNHRAQJWFU6Y2JIQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.707415294 (604h4m51.578498329s) [noise=true , address=145.79.227.50:4384 , piece_id=U5R4PT66AXFJFPAGFADCVX6NNNIFLFM3CFTREKVEOOUDHTVTUTBA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.732565163 (604h4m51.603648198s) [noise=true , address=145.79.227.36:4546 , piece_id=4V7GWX3MULB6CR3QH44ZIOISDONXKJ334XFSYHXUECJZBMRHQCKA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.745717487 (604h4m51.616800522s) [noise=true , address=64.49.13.170:4717 , piece_id=NSYCFF26VLJ6QPF5JAPBZRBLULLDAWUEPHI5HZTPSXQVX43KQHVQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.782268034 (604h4m51.653351069s) [noise=true , address=64.49.13.138:4483 , piece_id=GRAGOWDG4I5TDHVNX75QJBTS7NIIRY3B73VWDO7DWNITNJXBAYWQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.856613327 (604h4m51.727696362s) [noise=true , address=145.79.227.52:4458 , piece_id=2XCSF7MRBBZBGEJ2GE63K25BO5DFB7DUSZ4KXMFRS5NOOHPSEJRA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.900758014 (604h4m51.771841059s) [noise=true , address=77.90.31.10:4534 , piece_id=KPMW47ZBIINLCZ67FH6EGEDFLEAT7F35CYP62D4TT7IBLQ57I2FA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.968806577 (604h4m51.839889622s) [noise=true , address=77.90.31.13:4631 , piece_id=IDYOBFP732AUWTYULX2I5ETBTS3DFAVW2EVEZ4BB5NYZFA4P2FTA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.000820185 (604h4m51.87190323s) [noise=true , address=45.91.116.14:4199 , piece_id=J3LDA3N2Q6B3VTWS5LG25LUD5E2L7I3FD56JPOVJVCJYLXC3AKXQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.111919172 (604h4m51.983002207s) [noise=true , address=145.79.227.34:4908 , piece_id=X57UEZNRHXKLR7R3VEWRKCALEHCMTE7NR73QGALOEZ7NKOBKDDFA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.513345511 (604h4m52.384428556s) [noise=true , address=5.231.73.4:4027 , piece_id=DVNFM76WSUGT65ZKJY4RFVXXLZIDQJ7TQ5CYBPUMZ2GXQWJOIY3Q, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.525239208 (604h4m52.396322253s) [noise=true , address=145.79.227.49:4362 , piece_id=UXLOT5FU522U73SMQEUIZ4K43WTQEAV5XRVSGG6EWQON43UWQAYQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.547688835 (604h4m52.41877187s) [noise=true , address=145.79.227.30:4793 , piece_id=37HWPUMM2Q4CZDYKXZIKL5KKPEH4O6ASHI675HEP2K7EOCGBVLFA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.777085496 (604h4m52.648168531s) [noise=true , address=145.79.227.33:4887 , piece_id=I7CHA6ZOBOS7J4SHID6MQLGAVR66BZ2D2J4VSTFCQW525BPN5KHQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.777110584 (604h4m52.648193619s) [seg_pos={0 0}, upload=529467]: Segment reached optimal threshold of 14 pieces. Dec 30 19:33:52.777320053 (604h4m52.648403088s) [noise=true , address=45.91.116.9:4288 , piece_id=QLRNLFOW2RZ6JLJDNJEKBQ6JWJLOZ57NBPADF3KYVLL4L5FTC62A, seg_pos={0 0}, upload=529467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX): protocol: write tcp 188.166.225.170:38490->45.91.116.9:4288: use of closed network connection; write tcp 188.166.225.170:38490->45.91.116.9:4288: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38490->45.91.116.9:4288: use of closed network connection Dec 30 19:33:52.777326334 (604h4m52.64840938s) [noise=true , address=45.91.116.9:4288 , piece_id=QLRNLFOW2RZ6JLJDNJEKBQ6JWJLOZ57NBPADF3KYVLL4L5FTC62A, seg_pos={0 0}, upload=529467]: Long tail cancellation due to optimal threshold Dec 30 19:33:52.777615194 (604h4m52.648698229s) [noise=true , address=77.90.31.14:4669 , piece_id=O2Z45BXHUPJQZZ7EJG4FKS3MNDPHWPBT4EMZCZV2HTJSEVAI7WHQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T): protocol: expected piece hash; context canceled; EOF Dec 30 19:33:52.777622820 (604h4m52.648705855s) [noise=true , address=77.90.31.14:4669 , piece_id=O2Z45BXHUPJQZZ7EJG4FKS3MNDPHWPBT4EMZCZV2HTJSEVAI7WHQ, seg_pos={0 0}, upload=529467]: Long tail cancellation due to optimal threshold Dec 30 19:33:52.777648788 (604h4m52.648731823s) [noise=true , address=64.49.13.169:4708 , piece_id=BXCIK6V4R5LZHWUTJOCL66442ZM6T7IYWD2OH5OM2SM7GX5JBIZA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:36148->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:36148->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36148->64.49.13.169:4708: use of closed network connection Dec 30 19:33:52.777665260 (604h4m52.648748295s) [noise=true , address=64.49.13.169:4708 , piece_id=BXCIK6V4R5LZHWUTJOCL66442ZM6T7IYWD2OH5OM2SM7GX5JBIZA, seg_pos={0 0}, upload=529467]: Long tail cancellation due to optimal threshold Dec 30 19:33:52.777872405 (604h4m52.64895545s) [noise=true , address=145.79.227.4:4218 , piece_id=2H4DI7772BR4JLLGKJYTSKTKVOABDB4T6UJ74OBTLXHV23WPN5HA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mbzxg1BjmbXeHxGgheZmQoN91k8NgaAUUASPkJe3fEmS4eEkc): protocol: expected piece hash; context canceled; EOF Dec 30 19:33:52.777879939 (604h4m52.648962984s) [noise=true , address=145.79.227.4:4218 , piece_id=2H4DI7772BR4JLLGKJYTSKTKVOABDB4T6UJ74OBTLXHV23WPN5HA, seg_pos={0 0}, upload=529467]: Long tail cancellation due to optimal threshold Dec 30 19:33:52.777898815 (604h4m52.64898185s) [seg_pos={0 0}, upload=529467]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 19:33:52.777974709 (604h4m52.649057755s) [upload=529467]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:33:52.777979669 (604h4m52.649062714s) [upload=529467]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:33:53.082585408 (604h4m52.953668453s) [upload=529467]: Done uploading segments... Dec 30 19:33:53.082605185 (604h4m52.953688221s) [upload=529467]: Upload finished. err: <nil> Duration: 26.691888919s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X..X.....................X...............] duration: 30.419056729s trace: https://jaeger.dev.storj.io/trace/2e4bd3c87074131f FAIL (2025-12-30 19:45:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== 98432889s) [noise=true , address=145.79.227.49:4362 , piece_id=UXLOT5FU522U73SMQEUIZ4K43WTQEAV5XRVSGG6EWQON43UWQAYQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.827625579 (604h4m49.698708614s) [noise=true , address=145.79.227.4:4218 , piece_id=2H4DI7772BR4JLLGKJYTSKTKVOABDB4T6UJ74OBTLXHV23WPN5HA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.827942963 (604h4m49.699025998s) [noise=true , address=45.91.116.9:4288 , piece_id=QLRNLFOW2RZ6JLJDNJEKBQ6JWJLOZ57NBPADF3KYVLL4L5FTC62A, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.828225852 (604h4m49.699308887s) [noise=true , address=45.91.116.14:4199 , piece_id=J3LDA3N2Q6B3VTWS5LG25LUD5E2L7I3FD56JPOVJVCJYLXC3AKXQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.828502729 (604h4m49.699585764s) [noise=true , address=64.49.13.138:4483 , piece_id=GRAGOWDG4I5TDHVNX75QJBTS7NIIRY3B73VWDO7DWNITNJXBAYWQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.828558776 (604h4m49.699641821s) [noise=true , address=64.49.13.169:4708 , piece_id=BXCIK6V4R5LZHWUTJOCL66442ZM6T7IYWD2OH5OM2SM7GX5JBIZA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.828853557 (604h4m49.699936592s) [noise=true , address=145.79.227.50:4384 , piece_id=U5R4PT66AXFJFPAGFADCVX6NNNIFLFM3CFTREKVEOOUDHTVTUTBA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.829160732 (604h4m49.700243767s) [noise=true , address=5.231.73.4:4027 , piece_id=DVNFM76WSUGT65ZKJY4RFVXXLZIDQJ7TQ5CYBPUMZ2GXQWJOIY3Q, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.829534374 (604h4m49.700617409s) [noise=true , address=145.79.227.52:4458 , piece_id=2XCSF7MRBBZBGEJ2GE63K25BO5DFB7DUSZ4KXMFRS5NOOHPSEJRA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.829873299 (604h4m49.700956334s) [noise=true , address=145.79.227.34:4908 , piece_id=X57UEZNRHXKLR7R3VEWRKCALEHCMTE7NR73QGALOEZ7NKOBKDDFA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.830200823 (604h4m49.701283868s) [noise=true , address=145.79.227.33:4887 , piece_id=I7CHA6ZOBOS7J4SHID6MQLGAVR66BZ2D2J4VSTFCQW525BPN5KHQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.838625865 (604h4m49.7097089s) [noise=true , address=145.79.227.3:4180 , piece_id=CYOSWRJR37YSHNJBWZZPMOLM7TVRJ2GFEV4XNNHRAQJWFU6Y2JIQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.839781686 (604h4m49.710864731s) [noise=true , address=145.79.227.36:4546 , piece_id=4V7GWX3MULB6CR3QH44ZIOISDONXKJ334XFSYHXUECJZBMRHQCKA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.840846864 (604h4m49.711929899s) [noise=true , address=77.90.31.14:4669 , piece_id=O2Z45BXHUPJQZZ7EJG4FKS3MNDPHWPBT4EMZCZV2HTJSEVAI7WHQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.841958571 (604h4m49.713041606s) [noise=true , address=77.90.31.10:4534 , piece_id=KPMW47ZBIINLCZ67FH6EGEDFLEAT7F35CYP62D4TT7IBLQ57I2FA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:51.267701872 (604h4m51.138784917s) [upload=529467]: Next returned nil segment Dec 30 19:33:51.267719867 (604h4m51.138802912s) [upload=529467]: Waiting for error group managing segments... Dec 30 19:33:51.700925266 (604h4m51.572008302s) [noise=true , address=145.79.227.3:4180 , piece_id=CYOSWRJR37YSHNJBWZZPMOLM7TVRJ2GFEV4XNNHRAQJWFU6Y2JIQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.707415294 (604h4m51.578498329s) [noise=true , address=145.79.227.50:4384 , piece_id=U5R4PT66AXFJFPAGFADCVX6NNNIFLFM3CFTREKVEOOUDHTVTUTBA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.732565163 (604h4m51.603648198s) [noise=true , address=145.79.227.36:4546 , piece_id=4V7GWX3MULB6CR3QH44ZIOISDONXKJ334XFSYHXUECJZBMRHQCKA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.745717487 (604h4m51.616800522s) [noise=true , address=64.49.13.170:4717 , piece_id=NSYCFF26VLJ6QPF5JAPBZRBLULLDAWUEPHI5HZTPSXQVX43KQHVQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.782268034 (604h4m51.653351069s) [noise=true , address=64.49.13.138:4483 , piece_id=GRAGOWDG4I5TDHVNX75QJBTS7NIIRY3B73VWDO7DWNITNJXBAYWQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.856613327 (604h4m51.727696362s) [noise=true , address=145.79.227.52:4458 , piece_id=2XCSF7MRBBZBGEJ2GE63K25BO5DFB7DUSZ4KXMFRS5NOOHPSEJRA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.900758014 (604h4m51.771841059s) [noise=true , address=77.90.31.10:4534 , piece_id=KPMW47ZBIINLCZ67FH6EGEDFLEAT7F35CYP62D4TT7IBLQ57I2FA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.968806577 (604h4m51.839889622s) [noise=true , address=77.90.31.13:4631 , piece_id=IDYOBFP732AUWTYULX2I5ETBTS3DFAVW2EVEZ4BB5NYZFA4P2FTA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.000820185 (604h4m51.87190323s) [noise=true , address=45.91.116.14:4199 , piece_id=J3LDA3N2Q6B3VTWS5LG25LUD5E2L7I3FD56JPOVJVCJYLXC3AKXQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.111919172 (604h4m51.983002207s) [noise=true , address=145.79.227.34:4908 , piece_id=X57UEZNRHXKLR7R3VEWRKCALEHCMTE7NR73QGALOEZ7NKOBKDDFA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.513345511 (604h4m52.384428556s) [noise=true , address=5.231.73.4:4027 , piece_id=DVNFM76WSUGT65ZKJY4RFVXXLZIDQJ7TQ5CYBPUMZ2GXQWJOIY3Q, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.525239208 (604h4m52.396322253s) [noise=true , address=145.79.227.49:4362 , piece_id=UXLOT5FU522U73SMQEUIZ4K43WTQEAV5XRVSGG6EWQON43UWQAYQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.547688835 (604h4m52.41877187s) [noise=true , address=145.79.227.30:4793 , piece_id=37HWPUMM2Q4CZDYKXZIKL5KKPEH4O6ASHI675HEP2K7EOCGBVLFA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.777085496 (604h4m52.648168531s) [noise=true , address=145.79.227.33:4887 , piece_id=I7CHA6ZOBOS7J4SHID6MQLGAVR66BZ2D2J4VSTFCQW525BPN5KHQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.777110584 (604h4m52.648193619s) [seg_pos={0 0}, upload=529467]: Segment reached optimal threshold of 14 pieces. Dec 30 19:33:52.777320053 (604h4m52.648403088s) [noise=true , address=45.91.116.9:4288 , piece_id=QLRNLFOW2RZ6JLJDNJEKBQ6JWJLOZ57NBPADF3KYVLL4L5FTC62A, seg_pos={0 0}, upload=529467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX): protocol: write tcp 188.166.225.170:38490->45.91.116.9:4288: use of closed network connection; write tcp 188.166.225.170:38490->45.91.116.9:4288: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38490->45.91.116.9:4288: use of closed network connection Dec 30 19:33:52.777326334 (604h4m52.64840938s) [noise=true , address=45.91.116.9:4288 , piece_id=QLRNLFOW2RZ6JLJDNJEKBQ6JWJLOZ57NBPADF3KYVLL4L5FTC62A, seg_pos={0 0}, upload=529467]: Long tail cancellation due to optimal threshold Dec 30 19:33:52.777615194 (604h4m52.648698229s) [noise=true , address=77.90.31.14:4669 , piece_id=O2Z45BXHUPJQZZ7EJG4FKS3MNDPHWPBT4EMZCZV2HTJSEVAI7WHQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T): protocol: expected piece hash; context canceled; EOF Dec 30 19:33:52.777622820 (604h4m52.648705855s) [noise=true , address=77.90.31.14:4669 , piece_id=O2Z45BXHUPJQZZ7EJG4FKS3MNDPHWPBT4EMZCZV2HTJSEVAI7WHQ, seg_pos={0 0}, upload=529467]: Long tail cancellation due to optimal threshold Dec 30 19:33:52.777648788 (604h4m52.648731823s) [noise=true , address=64.49.13.169:4708 , piece_id=BXCIK6V4R5LZHWUTJOCL66442ZM6T7IYWD2OH5OM2SM7GX5JBIZA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:36148->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:36148->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36148->64.49.13.169:4708: use of closed network connection Dec 30 19:33:52.777665260 (604h4m52.648748295s) [noise=true , address=64.49.13.169:4708 , piece_id=BXCIK6V4R5LZHWUTJOCL66442ZM6T7IYWD2OH5OM2SM7GX5JBIZA, seg_pos={0 0}, upload=529467]: Long tail cancellation due to optimal threshold Dec 30 19:33:52.777872405 (604h4m52.64895545s) [noise=true , address=145.79.227.4:4218 , piece_id=2H4DI7772BR4JLLGKJYTSKTKVOABDB4T6UJ74OBTLXHV23WPN5HA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mbzxg1BjmbXeHxGgheZmQoN91k8NgaAUUASPkJe3fEmS4eEkc): protocol: expected piece hash; context canceled; EOF Dec 30 19:33:52.777879939 (604h4m52.648962984s) [noise=true , address=145.79.227.4:4218 , piece_id=2H4DI7772BR4JLLGKJYTSKTKVOABDB4T6UJ74OBTLXHV23WPN5HA, seg_pos={0 0}, upload=529467]: Long tail cancellation due to optimal threshold Dec 30 19:33:52.777898815 (604h4m52.64898185s) [seg_pos={0 0}, upload=529467]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 19:33:52.777974709 (604h4m52.649057755s) [upload=529467]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:33:52.777979669 (604h4m52.649062714s) [upload=529467]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:33:53.082585408 (604h4m52.953668453s) [upload=529467]: Done uploading segments... Dec 30 19:33:53.082605185 (604h4m52.953688221s) [upload=529467]: Upload finished. err: <nil> Duration: 30.419088781s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [....X..X..X.....X..XX...X..X.....XX.X.....X.....X...........] duration: 6.13794499s trace: https://jaeger.dev.storj.io/trace/4b57a4d55c27f17d FAIL (2025-12-30 19:45:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 2MXNCFG5BZSBRLTW4OPJXJWAVDPMHK4GXYPWT6GBFIH65F3IEMVA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: UU2AYD5D3SDGOW7JMKI6CV2SNPTVZNKJUW4TFNWSQ22WH4LAOYTQ) context deadline exceeded ======LOGS====== Dec 30 19:37:18.017831880 (604h8m17.888914925s) [upload=529518]: Starting upload Dec 30 19:37:18.017920869 (604h8m17.889003914s) [upload=529518]: Done starting upload Dec 30 19:37:18.017989800 (604h8m17.889072835s) [upload=529518]: Uploading segments... Dec 30 19:37:18.018167178 (604h8m17.889250213s) [upload=529518]: Got next segment. Inline: false Dec 30 19:37:18.018211322 (604h8m17.889294367s) [upload=529518]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 19:37:18.018216993 (604h8m17.889300028s) [upload=529518]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 19:37:18.265194540 (604h8m18.136277575s) [seg_pos={0 0}, upload=529518]: Begin upload segment... Dec 30 19:37:18.265242221 (604h8m18.136325267s) [seg_pos={0 0}, upload=529518]: Done begin upload segment. Dec 30 19:37:18.265271887 (604h8m18.136354922s) [noise=true , address=45.91.116.13:4156 , piece_id=ZXMVGKUW2Q4SQNBGD5ZZP7CHF2RFVVLSZIYEMVOOZB2VU4CEU6UQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.265280745 (604h8m18.13636378s) [noise=true , address=145.79.227.22:4510 , piece_id=TFAYL6XXPFSLYYPUK5GRBO7FDDDAJRMPGI2PTD3SH7OLW6XR5QGQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.265996978 (604h8m18.137080013s) [noise=true , address=145.79.227.13:4803 , piece_id=DGVTZAR4T33Z4ZWIRBWKLPTY45W7CDF5SPJ7IOSUBFLNQWAD4XOQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.266478024 (604h8m18.137561059s) [noise=true , address=45.91.116.8:4048 , piece_id=2MXNCFG5BZSBRLTW4OPJXJWAVDPMHK4GXYPWT6GBFIH65F3IEMVA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.267063399 (604h8m18.138146444s) [noise=true , address=145.79.227.12:4765 , piece_id=D4M33JNPTNF5EOTP6V4CH2KBZOJBKIRZ5RCEUMJMFSB5KEO3EITA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.267602044 (604h8m18.138685079s) [noise=true , address=45.91.116.11:4118 , piece_id=UU2AYD5D3SDGOW7JMKI6CV2SNPTVZNKJUW4TFNWSQ22WH4LAOYTQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.268103920 (604h8m18.139186965s) [noise=true , address=145.79.227.24:4582 , piece_id=DWQXXGUEQ4IXJJXA57AIULGBQTCM5LMRS6IKSDKNVDXIBAS3HMRA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.268624721 (604h8m18.139707756s) [noise=true , address=145.79.227.2:4114 , piece_id=5ELY6KB4QBNBOPMG6ALNYRCW3CYXRYVSDGMRAJYYYPE6TZSMA4OA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.269149150 (604h8m18.140232195s) [noise=true , address=45.91.116.7:4012 , piece_id=LOW2MEEFBV64QEH7MKQT2R4PHUBPWPRZGDYVCYMELUD4BOQTGEVA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.269680512 (604h8m18.140763547s) [noise=true , address=77.90.31.12:4589 , piece_id=RQH5BFFWHPXUA4NMBZUJYNOAI5OQU43HDCGKJMYBSS6O3YDFQL2A, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.270313487 (604h8m18.141396522s) [noise=true , address=145.79.227.46:4601 , piece_id=Y4BKLMSLP2SBMSKC3S6ISJCIRAXTZMIUXQDTRYXBRIEE6V4YQYLA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.270669034 (604h8m18.141752069s) [noise=true , address=45.91.116.12:4227 , piece_id=Q26IH3AZJJISAEL5LPYT7CRFTP4KUMXOX44MYSG3BCLZNLMYLERQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.270947995 (604h8m18.14203103s) [noise=true , address=145.79.227.23:4547 , piece_id=6G7N6QNXE3HHSWO6BMDRCQS26RJKV7PETS3VCAPLPCJNFHR4WGEA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.271235693 (604h8m18.142318738s) [noise=true , address=77.90.31.21:4880 , piece_id=2EIJYMPCZP4BWQX4I3RSQS5IDOEAMX7UAGQIXEVRJQG6MPBZSFQA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.271512129 (604h8m18.142595174s) [noise=true , address=145.79.227.51:4410 , piece_id=65NLCMFGMV2NLVDAYRXQMYVQDBAWQT4TUAFRIYDUZOGT46S5KYGA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.271809736 (604h8m18.142892782s) [noise=true , address=145.79.227.29:4941 , piece_id=J2QTCA233FRN5E6KQXXIP767JHUL46VQJ7I76UTJEWVLO4UYAHFQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:19.968123940 (604h8m19.839206985s) [upload=529518]: Next returned nil segment Dec 30 19:37:19.968138839 (604h8m19.839221874s) [upload=529518]: Waiting for error group managing segments... Dec 30 19:37:20.240602244 (604h8m20.111685279s) [noise=true , address=145.79.227.13:4803 , piece_id=DGVTZAR4T33Z4ZWIRBWKLPTY45W7CDF5SPJ7IOSUBFLNQWAD4XOQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.242668559 (604h8m20.113751594s) [noise=true , address=145.79.227.2:4114 , piece_id=5ELY6KB4QBNBOPMG6ALNYRCW3CYXRYVSDGMRAJYYYPE6TZSMA4OA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.274323484 (604h8m20.145406529s) [noise=true , address=145.79.227.51:4410 , piece_id=65NLCMFGMV2NLVDAYRXQMYVQDBAWQT4TUAFRIYDUZOGT46S5KYGA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.276151625 (604h8m20.14723467s) [noise=true , address=145.79.227.24:4582 , piece_id=DWQXXGUEQ4IXJJXA57AIULGBQTCM5LMRS6IKSDKNVDXIBAS3HMRA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.325870757 (604h8m20.196953792s) [noise=true , address=145.79.227.46:4601 , piece_id=Y4BKLMSLP2SBMSKC3S6ISJCIRAXTZMIUXQDTRYXBRIEE6V4YQYLA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.338603552 (604h8m20.209686587s) [noise=true , address=145.79.227.22:4510 , piece_id=TFAYL6XXPFSLYYPUK5GRBO7FDDDAJRMPGI2PTD3SH7OLW6XR5QGQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.339426289 (604h8m20.210509334s) [noise=true , address=145.79.227.12:4765 , piece_id=D4M33JNPTNF5EOTP6V4CH2KBZOJBKIRZ5RCEUMJMFSB5KEO3EITA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.528772708 (604h8m20.399855963s) [noise=true , address=77.90.31.21:4880 , piece_id=2EIJYMPCZP4BWQX4I3RSQS5IDOEAMX7UAGQIXEVRJQG6MPBZSFQA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.723971783 (604h8m20.595054818s) [noise=true , address=45.91.116.12:4227 , piece_id=Q26IH3AZJJISAEL5LPYT7CRFTP4KUMXOX44MYSG3BCLZNLMYLERQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.738010033 (604h8m20.609093068s) [noise=true , address=45.91.116.7:4012 , piece_id=LOW2MEEFBV64QEH7MKQT2R4PHUBPWPRZGDYVCYMELUD4BOQTGEVA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.900656063 (604h8m20.771739098s) [noise=true , address=45.91.116.8:4048 , piece_id=2MXNCFG5BZSBRLTW4OPJXJWAVDPMHK4GXYPWT6GBFIH65F3IEMVA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:21.200841094 (604h8m21.07192414s) [noise=true , address=77.90.31.12:4589 , piece_id=RQH5BFFWHPXUA4NMBZUJYNOAI5OQU43HDCGKJMYBSS6O3YDFQL2A, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:21.316516896 (604h8m21.187599941s) [noise=true , address=45.91.116.11:4118 , piece_id=UU2AYD5D3SDGOW7JMKI6CV2SNPTVZNKJUW4TFNWSQ22WH4LAOYTQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:21.316538046 (604h8m21.187621081s) [seg_pos={0 0}, upload=529518]: Segment reached optimal threshold of 13 pieces. Dec 30 19:37:21.316716626 (604h8m21.187799661s) [noise=true , address=145.79.227.29:4941 , piece_id=J2QTCA233FRN5E6KQXXIP767JHUL46VQJ7I76UTJEWVLO4UYAHFQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 30 19:37:21.316723229 (604h8m21.187806264s) [noise=true , address=145.79.227.29:4941 , piece_id=J2QTCA233FRN5E6KQXXIP767JHUL46VQJ7I76UTJEWVLO4UYAHFQ, seg_pos={0 0}, upload=529518]: Long tail cancellation due to optimal threshold Dec 30 19:37:21.316823179 (604h8m21.187906214s) [noise=true , address=45.91.116.13:4156 , piece_id=ZXMVGKUW2Q4SQNBGD5ZZP7CHF2RFVVLSZIYEMVOOZB2VU4CEU6UQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 30 19:37:21.316831525 (604h8m21.18791456s) [noise=true , address=45.91.116.13:4156 , piece_id=ZXMVGKUW2Q4SQNBGD5ZZP7CHF2RFVVLSZIYEMVOOZB2VU4CEU6UQ, seg_pos={0 0}, upload=529518]: Long tail cancellation due to optimal threshold Dec 30 19:37:21.317004414 (604h8m21.18808745s) [noise=true , address=145.79.227.23:4547 , piece_id=6G7N6QNXE3HHSWO6BMDRCQS26RJKV7PETS3VCAPLPCJNFHR4WGEA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: write tcp 188.166.225.170:45998->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:45998->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45998->145.79.227.23:4547: use of closed network connection Dec 30 19:37:21.317013822 (604h8m21.188096877s) [noise=true , address=145.79.227.23:4547 , piece_id=6G7N6QNXE3HHSWO6BMDRCQS26RJKV7PETS3VCAPLPCJNFHR4WGEA, seg_pos={0 0}, upload=529518]: Long tail cancellation due to optimal threshold Dec 30 19:37:21.317033068 (604h8m21.188116113s) [seg_pos={0 0}, upload=529518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 19:37:21.317121227 (604h8m21.188204262s) [upload=529518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:37:21.317128490 (604h8m21.188211555s) [upload=529518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:37:21.616086595 (604h8m21.48716965s) [upload=529518]: Done uploading segments... Dec 30 19:37:21.616243293 (604h8m21.487326328s) [upload=529518]: Upload finished. err: <nil> Duration: 6.137996157s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.....X..X..X.....X..XX...X..X.....XX.X.....X.....X..........] duration: 6.391921632s trace: https://jaeger.dev.storj.io/trace/4b57a4d55c27f17d FAIL (2025-12-30 19:44:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 2MXNCFG5BZSBRLTW4OPJXJWAVDPMHK4GXYPWT6GBFIH65F3IEMVA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: UU2AYD5D3SDGOW7JMKI6CV2SNPTVZNKJUW4TFNWSQ22WH4LAOYTQ) context deadline exceeded ======LOGS====== Dec 30 19:37:18.017831880 (604h8m17.888914925s) [upload=529518]: Starting upload Dec 30 19:37:18.017920869 (604h8m17.889003914s) [upload=529518]: Done starting upload Dec 30 19:37:18.017989800 (604h8m17.889072835s) [upload=529518]: Uploading segments... Dec 30 19:37:18.018167178 (604h8m17.889250213s) [upload=529518]: Got next segment. Inline: false Dec 30 19:37:18.018211322 (604h8m17.889294367s) [upload=529518]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 19:37:18.018216993 (604h8m17.889300028s) [upload=529518]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 19:37:18.265194540 (604h8m18.136277575s) [seg_pos={0 0}, upload=529518]: Begin upload segment... Dec 30 19:37:18.265242221 (604h8m18.136325267s) [seg_pos={0 0}, upload=529518]: Done begin upload segment. Dec 30 19:37:18.265271887 (604h8m18.136354922s) [noise=true , address=45.91.116.13:4156 , piece_id=ZXMVGKUW2Q4SQNBGD5ZZP7CHF2RFVVLSZIYEMVOOZB2VU4CEU6UQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.265280745 (604h8m18.13636378s) [noise=true , address=145.79.227.22:4510 , piece_id=TFAYL6XXPFSLYYPUK5GRBO7FDDDAJRMPGI2PTD3SH7OLW6XR5QGQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.265996978 (604h8m18.137080013s) [noise=true , address=145.79.227.13:4803 , piece_id=DGVTZAR4T33Z4ZWIRBWKLPTY45W7CDF5SPJ7IOSUBFLNQWAD4XOQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.266478024 (604h8m18.137561059s) [noise=true , address=45.91.116.8:4048 , piece_id=2MXNCFG5BZSBRLTW4OPJXJWAVDPMHK4GXYPWT6GBFIH65F3IEMVA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.267063399 (604h8m18.138146444s) [noise=true , address=145.79.227.12:4765 , piece_id=D4M33JNPTNF5EOTP6V4CH2KBZOJBKIRZ5RCEUMJMFSB5KEO3EITA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.267602044 (604h8m18.138685079s) [noise=true , address=45.91.116.11:4118 , piece_id=UU2AYD5D3SDGOW7JMKI6CV2SNPTVZNKJUW4TFNWSQ22WH4LAOYTQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.268103920 (604h8m18.139186965s) [noise=true , address=145.79.227.24:4582 , piece_id=DWQXXGUEQ4IXJJXA57AIULGBQTCM5LMRS6IKSDKNVDXIBAS3HMRA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.268624721 (604h8m18.139707756s) [noise=true , address=145.79.227.2:4114 , piece_id=5ELY6KB4QBNBOPMG6ALNYRCW3CYXRYVSDGMRAJYYYPE6TZSMA4OA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.269149150 (604h8m18.140232195s) [noise=true , address=45.91.116.7:4012 , piece_id=LOW2MEEFBV64QEH7MKQT2R4PHUBPWPRZGDYVCYMELUD4BOQTGEVA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.269680512 (604h8m18.140763547s) [noise=true , address=77.90.31.12:4589 , piece_id=RQH5BFFWHPXUA4NMBZUJYNOAI5OQU43HDCGKJMYBSS6O3YDFQL2A, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.270313487 (604h8m18.141396522s) [noise=true , address=145.79.227.46:4601 , piece_id=Y4BKLMSLP2SBMSKC3S6ISJCIRAXTZMIUXQDTRYXBRIEE6V4YQYLA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.270669034 (604h8m18.141752069s) [noise=true , address=45.91.116.12:4227 , piece_id=Q26IH3AZJJISAEL5LPYT7CRFTP4KUMXOX44MYSG3BCLZNLMYLERQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.270947995 (604h8m18.14203103s) [noise=true , address=145.79.227.23:4547 , piece_id=6G7N6QNXE3HHSWO6BMDRCQS26RJKV7PETS3VCAPLPCJNFHR4WGEA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.271235693 (604h8m18.142318738s) [noise=true , address=77.90.31.21:4880 , piece_id=2EIJYMPCZP4BWQX4I3RSQS5IDOEAMX7UAGQIXEVRJQG6MPBZSFQA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.271512129 (604h8m18.142595174s) [noise=true , address=145.79.227.51:4410 , piece_id=65NLCMFGMV2NLVDAYRXQMYVQDBAWQT4TUAFRIYDUZOGT46S5KYGA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.271809736 (604h8m18.142892782s) [noise=true , address=145.79.227.29:4941 , piece_id=J2QTCA233FRN5E6KQXXIP767JHUL46VQJ7I76UTJEWVLO4UYAHFQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:19.968123940 (604h8m19.839206985s) [upload=529518]: Next returned nil segment Dec 30 19:37:19.968138839 (604h8m19.839221874s) [upload=529518]: Waiting for error group managing segments... Dec 30 19:37:20.240602244 (604h8m20.111685279s) [noise=true , address=145.79.227.13:4803 , piece_id=DGVTZAR4T33Z4ZWIRBWKLPTY45W7CDF5SPJ7IOSUBFLNQWAD4XOQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.242668559 (604h8m20.113751594s) [noise=true , address=145.79.227.2:4114 , piece_id=5ELY6KB4QBNBOPMG6ALNYRCW3CYXRYVSDGMRAJYYYPE6TZSMA4OA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.274323484 (604h8m20.145406529s) [noise=true , address=145.79.227.51:4410 , piece_id=65NLCMFGMV2NLVDAYRXQMYVQDBAWQT4TUAFRIYDUZOGT46S5KYGA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.276151625 (604h8m20.14723467s) [noise=true , address=145.79.227.24:4582 , piece_id=DWQXXGUEQ4IXJJXA57AIULGBQTCM5LMRS6IKSDKNVDXIBAS3HMRA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.325870757 (604h8m20.196953792s) [noise=true , address=145.79.227.46:4601 , piece_id=Y4BKLMSLP2SBMSKC3S6ISJCIRAXTZMIUXQDTRYXBRIEE6V4YQYLA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.338603552 (604h8m20.209686587s) [noise=true , address=145.79.227.22:4510 , piece_id=TFAYL6XXPFSLYYPUK5GRBO7FDDDAJRMPGI2PTD3SH7OLW6XR5QGQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.339426289 (604h8m20.210509334s) [noise=true , address=145.79.227.12:4765 , piece_id=D4M33JNPTNF5EOTP6V4CH2KBZOJBKIRZ5RCEUMJMFSB5KEO3EITA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.528772708 (604h8m20.399855963s) [noise=true , address=77.90.31.21:4880 , piece_id=2EIJYMPCZP4BWQX4I3RSQS5IDOEAMX7UAGQIXEVRJQG6MPBZSFQA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.723971783 (604h8m20.595054818s) [noise=true , address=45.91.116.12:4227 , piece_id=Q26IH3AZJJISAEL5LPYT7CRFTP4KUMXOX44MYSG3BCLZNLMYLERQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.738010033 (604h8m20.609093068s) [noise=true , address=45.91.116.7:4012 , piece_id=LOW2MEEFBV64QEH7MKQT2R4PHUBPWPRZGDYVCYMELUD4BOQTGEVA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.900656063 (604h8m20.771739098s) [noise=true , address=45.91.116.8:4048 , piece_id=2MXNCFG5BZSBRLTW4OPJXJWAVDPMHK4GXYPWT6GBFIH65F3IEMVA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:21.200841094 (604h8m21.07192414s) [noise=true , address=77.90.31.12:4589 , piece_id=RQH5BFFWHPXUA4NMBZUJYNOAI5OQU43HDCGKJMYBSS6O3YDFQL2A, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:21.316516896 (604h8m21.187599941s) [noise=true , address=45.91.116.11:4118 , piece_id=UU2AYD5D3SDGOW7JMKI6CV2SNPTVZNKJUW4TFNWSQ22WH4LAOYTQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:21.316538046 (604h8m21.187621081s) [seg_pos={0 0}, upload=529518]: Segment reached optimal threshold of 13 pieces. Dec 30 19:37:21.316716626 (604h8m21.187799661s) [noise=true , address=145.79.227.29:4941 , piece_id=J2QTCA233FRN5E6KQXXIP767JHUL46VQJ7I76UTJEWVLO4UYAHFQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 30 19:37:21.316723229 (604h8m21.187806264s) [noise=true , address=145.79.227.29:4941 , piece_id=J2QTCA233FRN5E6KQXXIP767JHUL46VQJ7I76UTJEWVLO4UYAHFQ, seg_pos={0 0}, upload=529518]: Long tail cancellation due to optimal threshold Dec 30 19:37:21.316823179 (604h8m21.187906214s) [noise=true , address=45.91.116.13:4156 , piece_id=ZXMVGKUW2Q4SQNBGD5ZZP7CHF2RFVVLSZIYEMVOOZB2VU4CEU6UQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 30 19:37:21.316831525 (604h8m21.18791456s) [noise=true , address=45.91.116.13:4156 , piece_id=ZXMVGKUW2Q4SQNBGD5ZZP7CHF2RFVVLSZIYEMVOOZB2VU4CEU6UQ, seg_pos={0 0}, upload=529518]: Long tail cancellation due to optimal threshold Dec 30 19:37:21.317004414 (604h8m21.18808745s) [noise=true , address=145.79.227.23:4547 , piece_id=6G7N6QNXE3HHSWO6BMDRCQS26RJKV7PETS3VCAPLPCJNFHR4WGEA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: write tcp 188.166.225.170:45998->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:45998->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45998->145.79.227.23:4547: use of closed network connection Dec 30 19:37:21.317013822 (604h8m21.188096877s) [noise=true , address=145.79.227.23:4547 , piece_id=6G7N6QNXE3HHSWO6BMDRCQS26RJKV7PETS3VCAPLPCJNFHR4WGEA, seg_pos={0 0}, upload=529518]: Long tail cancellation due to optimal threshold Dec 30 19:37:21.317033068 (604h8m21.188116113s) [seg_pos={0 0}, upload=529518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 19:37:21.317121227 (604h8m21.188204262s) [upload=529518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:37:21.317128490 (604h8m21.188211555s) [upload=529518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:37:21.616086595 (604h8m21.48716965s) [upload=529518]: Done uploading segments... Dec 30 19:37:21.616243293 (604h8m21.487326328s) [upload=529518]: Upload finished. err: <nil> Duration: 6.391946308s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X..X.....................X..............] duration: 17.145081056s trace: https://jaeger.dev.storj.io/trace/2e4bd3c87074131f FAIL (2025-12-30 19:44:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: quiescence ======LOGS====== 98432889s) [noise=true , address=145.79.227.49:4362 , piece_id=UXLOT5FU522U73SMQEUIZ4K43WTQEAV5XRVSGG6EWQON43UWQAYQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.827625579 (604h4m49.698708614s) [noise=true , address=145.79.227.4:4218 , piece_id=2H4DI7772BR4JLLGKJYTSKTKVOABDB4T6UJ74OBTLXHV23WPN5HA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.827942963 (604h4m49.699025998s) [noise=true , address=45.91.116.9:4288 , piece_id=QLRNLFOW2RZ6JLJDNJEKBQ6JWJLOZ57NBPADF3KYVLL4L5FTC62A, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.828225852 (604h4m49.699308887s) [noise=true , address=45.91.116.14:4199 , piece_id=J3LDA3N2Q6B3VTWS5LG25LUD5E2L7I3FD56JPOVJVCJYLXC3AKXQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.828502729 (604h4m49.699585764s) [noise=true , address=64.49.13.138:4483 , piece_id=GRAGOWDG4I5TDHVNX75QJBTS7NIIRY3B73VWDO7DWNITNJXBAYWQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.828558776 (604h4m49.699641821s) [noise=true , address=64.49.13.169:4708 , piece_id=BXCIK6V4R5LZHWUTJOCL66442ZM6T7IYWD2OH5OM2SM7GX5JBIZA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.828853557 (604h4m49.699936592s) [noise=true , address=145.79.227.50:4384 , piece_id=U5R4PT66AXFJFPAGFADCVX6NNNIFLFM3CFTREKVEOOUDHTVTUTBA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.829160732 (604h4m49.700243767s) [noise=true , address=5.231.73.4:4027 , piece_id=DVNFM76WSUGT65ZKJY4RFVXXLZIDQJ7TQ5CYBPUMZ2GXQWJOIY3Q, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.829534374 (604h4m49.700617409s) [noise=true , address=145.79.227.52:4458 , piece_id=2XCSF7MRBBZBGEJ2GE63K25BO5DFB7DUSZ4KXMFRS5NOOHPSEJRA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.829873299 (604h4m49.700956334s) [noise=true , address=145.79.227.34:4908 , piece_id=X57UEZNRHXKLR7R3VEWRKCALEHCMTE7NR73QGALOEZ7NKOBKDDFA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.830200823 (604h4m49.701283868s) [noise=true , address=145.79.227.33:4887 , piece_id=I7CHA6ZOBOS7J4SHID6MQLGAVR66BZ2D2J4VSTFCQW525BPN5KHQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.838625865 (604h4m49.7097089s) [noise=true , address=145.79.227.3:4180 , piece_id=CYOSWRJR37YSHNJBWZZPMOLM7TVRJ2GFEV4XNNHRAQJWFU6Y2JIQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.839781686 (604h4m49.710864731s) [noise=true , address=145.79.227.36:4546 , piece_id=4V7GWX3MULB6CR3QH44ZIOISDONXKJ334XFSYHXUECJZBMRHQCKA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.840846864 (604h4m49.711929899s) [noise=true , address=77.90.31.14:4669 , piece_id=O2Z45BXHUPJQZZ7EJG4FKS3MNDPHWPBT4EMZCZV2HTJSEVAI7WHQ, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:49.841958571 (604h4m49.713041606s) [noise=true , address=77.90.31.10:4534 , piece_id=KPMW47ZBIINLCZ67FH6EGEDFLEAT7F35CYP62D4TT7IBLQ57I2FA, seg_pos={0 0}, upload=529467]: Uploading piece... Dec 30 19:33:51.267701872 (604h4m51.138784917s) [upload=529467]: Next returned nil segment Dec 30 19:33:51.267719867 (604h4m51.138802912s) [upload=529467]: Waiting for error group managing segments... Dec 30 19:33:51.700925266 (604h4m51.572008302s) [noise=true , address=145.79.227.3:4180 , piece_id=CYOSWRJR37YSHNJBWZZPMOLM7TVRJ2GFEV4XNNHRAQJWFU6Y2JIQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.707415294 (604h4m51.578498329s) [noise=true , address=145.79.227.50:4384 , piece_id=U5R4PT66AXFJFPAGFADCVX6NNNIFLFM3CFTREKVEOOUDHTVTUTBA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.732565163 (604h4m51.603648198s) [noise=true , address=145.79.227.36:4546 , piece_id=4V7GWX3MULB6CR3QH44ZIOISDONXKJ334XFSYHXUECJZBMRHQCKA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.745717487 (604h4m51.616800522s) [noise=true , address=64.49.13.170:4717 , piece_id=NSYCFF26VLJ6QPF5JAPBZRBLULLDAWUEPHI5HZTPSXQVX43KQHVQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.782268034 (604h4m51.653351069s) [noise=true , address=64.49.13.138:4483 , piece_id=GRAGOWDG4I5TDHVNX75QJBTS7NIIRY3B73VWDO7DWNITNJXBAYWQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.856613327 (604h4m51.727696362s) [noise=true , address=145.79.227.52:4458 , piece_id=2XCSF7MRBBZBGEJ2GE63K25BO5DFB7DUSZ4KXMFRS5NOOHPSEJRA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.900758014 (604h4m51.771841059s) [noise=true , address=77.90.31.10:4534 , piece_id=KPMW47ZBIINLCZ67FH6EGEDFLEAT7F35CYP62D4TT7IBLQ57I2FA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:51.968806577 (604h4m51.839889622s) [noise=true , address=77.90.31.13:4631 , piece_id=IDYOBFP732AUWTYULX2I5ETBTS3DFAVW2EVEZ4BB5NYZFA4P2FTA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.000820185 (604h4m51.87190323s) [noise=true , address=45.91.116.14:4199 , piece_id=J3LDA3N2Q6B3VTWS5LG25LUD5E2L7I3FD56JPOVJVCJYLXC3AKXQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.111919172 (604h4m51.983002207s) [noise=true , address=145.79.227.34:4908 , piece_id=X57UEZNRHXKLR7R3VEWRKCALEHCMTE7NR73QGALOEZ7NKOBKDDFA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.513345511 (604h4m52.384428556s) [noise=true , address=5.231.73.4:4027 , piece_id=DVNFM76WSUGT65ZKJY4RFVXXLZIDQJ7TQ5CYBPUMZ2GXQWJOIY3Q, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.525239208 (604h4m52.396322253s) [noise=true , address=145.79.227.49:4362 , piece_id=UXLOT5FU522U73SMQEUIZ4K43WTQEAV5XRVSGG6EWQON43UWQAYQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.547688835 (604h4m52.41877187s) [noise=true , address=145.79.227.30:4793 , piece_id=37HWPUMM2Q4CZDYKXZIKL5KKPEH4O6ASHI675HEP2K7EOCGBVLFA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.777085496 (604h4m52.648168531s) [noise=true , address=145.79.227.33:4887 , piece_id=I7CHA6ZOBOS7J4SHID6MQLGAVR66BZ2D2J4VSTFCQW525BPN5KHQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: <nil> Dec 30 19:33:52.777110584 (604h4m52.648193619s) [seg_pos={0 0}, upload=529467]: Segment reached optimal threshold of 14 pieces. Dec 30 19:33:52.777320053 (604h4m52.648403088s) [noise=true , address=45.91.116.9:4288 , piece_id=QLRNLFOW2RZ6JLJDNJEKBQ6JWJLOZ57NBPADF3KYVLL4L5FTC62A, seg_pos={0 0}, upload=529467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1H1XywjaRt5ZRGqmZBgzfSowuCnPqFesL1Ch2jYYS1gV2oBgQX): protocol: write tcp 188.166.225.170:38490->45.91.116.9:4288: use of closed network connection; write tcp 188.166.225.170:38490->45.91.116.9:4288: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38490->45.91.116.9:4288: use of closed network connection Dec 30 19:33:52.777326334 (604h4m52.64840938s) [noise=true , address=45.91.116.9:4288 , piece_id=QLRNLFOW2RZ6JLJDNJEKBQ6JWJLOZ57NBPADF3KYVLL4L5FTC62A, seg_pos={0 0}, upload=529467]: Long tail cancellation due to optimal threshold Dec 30 19:33:52.777615194 (604h4m52.648698229s) [noise=true , address=77.90.31.14:4669 , piece_id=O2Z45BXHUPJQZZ7EJG4FKS3MNDPHWPBT4EMZCZV2HTJSEVAI7WHQ, seg_pos={0 0}, upload=529467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgv8uy7FaviWBw3H8ZS2NfJNnZ5F5PYyxxJrYaJsfGVv55Q8T): protocol: expected piece hash; context canceled; EOF Dec 30 19:33:52.777622820 (604h4m52.648705855s) [noise=true , address=77.90.31.14:4669 , piece_id=O2Z45BXHUPJQZZ7EJG4FKS3MNDPHWPBT4EMZCZV2HTJSEVAI7WHQ, seg_pos={0 0}, upload=529467]: Long tail cancellation due to optimal threshold Dec 30 19:33:52.777648788 (604h4m52.648731823s) [noise=true , address=64.49.13.169:4708 , piece_id=BXCIK6V4R5LZHWUTJOCL66442ZM6T7IYWD2OH5OM2SM7GX5JBIZA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CTzrehDEVX9VJdfpPHUwmixehj18F1hsPLhifikqU9B3Gt3Hy): protocol: write tcp 188.166.225.170:36148->64.49.13.169:4708: use of closed network connection; write tcp 188.166.225.170:36148->64.49.13.169:4708: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36148->64.49.13.169:4708: use of closed network connection Dec 30 19:33:52.777665260 (604h4m52.648748295s) [noise=true , address=64.49.13.169:4708 , piece_id=BXCIK6V4R5LZHWUTJOCL66442ZM6T7IYWD2OH5OM2SM7GX5JBIZA, seg_pos={0 0}, upload=529467]: Long tail cancellation due to optimal threshold Dec 30 19:33:52.777872405 (604h4m52.64895545s) [noise=true , address=145.79.227.4:4218 , piece_id=2H4DI7772BR4JLLGKJYTSKTKVOABDB4T6UJ74OBTLXHV23WPN5HA, seg_pos={0 0}, upload=529467]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mbzxg1BjmbXeHxGgheZmQoN91k8NgaAUUASPkJe3fEmS4eEkc): protocol: expected piece hash; context canceled; EOF Dec 30 19:33:52.777879939 (604h4m52.648962984s) [noise=true , address=145.79.227.4:4218 , piece_id=2H4DI7772BR4JLLGKJYTSKTKVOABDB4T6UJ74OBTLXHV23WPN5HA, seg_pos={0 0}, upload=529467]: Long tail cancellation due to optimal threshold Dec 30 19:33:52.777898815 (604h4m52.64898185s) [seg_pos={0 0}, upload=529467]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 19:33:52.777974709 (604h4m52.649057755s) [upload=529467]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:33:52.777979669 (604h4m52.649062714s) [upload=529467]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:33:53.082585408 (604h4m52.953668453s) [upload=529467]: Done uploading segments... Dec 30 19:33:53.082605185 (604h4m52.953688221s) [upload=529467]: Upload finished. err: <nil> Duration: 17.145108859s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [......X..X..X.....X..XX...X..X.....XX.X.....X.....X.........] duration: 27.376175626s trace: https://jaeger.dev.storj.io/trace/4b57a4d55c27f17d FAIL (2025-12-30 19:44:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 2MXNCFG5BZSBRLTW4OPJXJWAVDPMHK4GXYPWT6GBFIH65F3IEMVA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: UU2AYD5D3SDGOW7JMKI6CV2SNPTVZNKJUW4TFNWSQ22WH4LAOYTQ) context deadline exceeded ======LOGS====== Dec 30 19:37:18.017831880 (604h8m17.888914925s) [upload=529518]: Starting upload Dec 30 19:37:18.017920869 (604h8m17.889003914s) [upload=529518]: Done starting upload Dec 30 19:37:18.017989800 (604h8m17.889072835s) [upload=529518]: Uploading segments... Dec 30 19:37:18.018167178 (604h8m17.889250213s) [upload=529518]: Got next segment. Inline: false Dec 30 19:37:18.018211322 (604h8m17.889294367s) [upload=529518]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 19:37:18.018216993 (604h8m17.889300028s) [upload=529518]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 19:37:18.265194540 (604h8m18.136277575s) [seg_pos={0 0}, upload=529518]: Begin upload segment... Dec 30 19:37:18.265242221 (604h8m18.136325267s) [seg_pos={0 0}, upload=529518]: Done begin upload segment. Dec 30 19:37:18.265271887 (604h8m18.136354922s) [noise=true , address=45.91.116.13:4156 , piece_id=ZXMVGKUW2Q4SQNBGD5ZZP7CHF2RFVVLSZIYEMVOOZB2VU4CEU6UQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.265280745 (604h8m18.13636378s) [noise=true , address=145.79.227.22:4510 , piece_id=TFAYL6XXPFSLYYPUK5GRBO7FDDDAJRMPGI2PTD3SH7OLW6XR5QGQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.265996978 (604h8m18.137080013s) [noise=true , address=145.79.227.13:4803 , piece_id=DGVTZAR4T33Z4ZWIRBWKLPTY45W7CDF5SPJ7IOSUBFLNQWAD4XOQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.266478024 (604h8m18.137561059s) [noise=true , address=45.91.116.8:4048 , piece_id=2MXNCFG5BZSBRLTW4OPJXJWAVDPMHK4GXYPWT6GBFIH65F3IEMVA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.267063399 (604h8m18.138146444s) [noise=true , address=145.79.227.12:4765 , piece_id=D4M33JNPTNF5EOTP6V4CH2KBZOJBKIRZ5RCEUMJMFSB5KEO3EITA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.267602044 (604h8m18.138685079s) [noise=true , address=45.91.116.11:4118 , piece_id=UU2AYD5D3SDGOW7JMKI6CV2SNPTVZNKJUW4TFNWSQ22WH4LAOYTQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.268103920 (604h8m18.139186965s) [noise=true , address=145.79.227.24:4582 , piece_id=DWQXXGUEQ4IXJJXA57AIULGBQTCM5LMRS6IKSDKNVDXIBAS3HMRA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.268624721 (604h8m18.139707756s) [noise=true , address=145.79.227.2:4114 , piece_id=5ELY6KB4QBNBOPMG6ALNYRCW3CYXRYVSDGMRAJYYYPE6TZSMA4OA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.269149150 (604h8m18.140232195s) [noise=true , address=45.91.116.7:4012 , piece_id=LOW2MEEFBV64QEH7MKQT2R4PHUBPWPRZGDYVCYMELUD4BOQTGEVA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.269680512 (604h8m18.140763547s) [noise=true , address=77.90.31.12:4589 , piece_id=RQH5BFFWHPXUA4NMBZUJYNOAI5OQU43HDCGKJMYBSS6O3YDFQL2A, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.270313487 (604h8m18.141396522s) [noise=true , address=145.79.227.46:4601 , piece_id=Y4BKLMSLP2SBMSKC3S6ISJCIRAXTZMIUXQDTRYXBRIEE6V4YQYLA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.270669034 (604h8m18.141752069s) [noise=true , address=45.91.116.12:4227 , piece_id=Q26IH3AZJJISAEL5LPYT7CRFTP4KUMXOX44MYSG3BCLZNLMYLERQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.270947995 (604h8m18.14203103s) [noise=true , address=145.79.227.23:4547 , piece_id=6G7N6QNXE3HHSWO6BMDRCQS26RJKV7PETS3VCAPLPCJNFHR4WGEA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.271235693 (604h8m18.142318738s) [noise=true , address=77.90.31.21:4880 , piece_id=2EIJYMPCZP4BWQX4I3RSQS5IDOEAMX7UAGQIXEVRJQG6MPBZSFQA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.271512129 (604h8m18.142595174s) [noise=true , address=145.79.227.51:4410 , piece_id=65NLCMFGMV2NLVDAYRXQMYVQDBAWQT4TUAFRIYDUZOGT46S5KYGA, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:18.271809736 (604h8m18.142892782s) [noise=true , address=145.79.227.29:4941 , piece_id=J2QTCA233FRN5E6KQXXIP767JHUL46VQJ7I76UTJEWVLO4UYAHFQ, seg_pos={0 0}, upload=529518]: Uploading piece... Dec 30 19:37:19.968123940 (604h8m19.839206985s) [upload=529518]: Next returned nil segment Dec 30 19:37:19.968138839 (604h8m19.839221874s) [upload=529518]: Waiting for error group managing segments... Dec 30 19:37:20.240602244 (604h8m20.111685279s) [noise=true , address=145.79.227.13:4803 , piece_id=DGVTZAR4T33Z4ZWIRBWKLPTY45W7CDF5SPJ7IOSUBFLNQWAD4XOQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.242668559 (604h8m20.113751594s) [noise=true , address=145.79.227.2:4114 , piece_id=5ELY6KB4QBNBOPMG6ALNYRCW3CYXRYVSDGMRAJYYYPE6TZSMA4OA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.274323484 (604h8m20.145406529s) [noise=true , address=145.79.227.51:4410 , piece_id=65NLCMFGMV2NLVDAYRXQMYVQDBAWQT4TUAFRIYDUZOGT46S5KYGA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.276151625 (604h8m20.14723467s) [noise=true , address=145.79.227.24:4582 , piece_id=DWQXXGUEQ4IXJJXA57AIULGBQTCM5LMRS6IKSDKNVDXIBAS3HMRA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.325870757 (604h8m20.196953792s) [noise=true , address=145.79.227.46:4601 , piece_id=Y4BKLMSLP2SBMSKC3S6ISJCIRAXTZMIUXQDTRYXBRIEE6V4YQYLA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.338603552 (604h8m20.209686587s) [noise=true , address=145.79.227.22:4510 , piece_id=TFAYL6XXPFSLYYPUK5GRBO7FDDDAJRMPGI2PTD3SH7OLW6XR5QGQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.339426289 (604h8m20.210509334s) [noise=true , address=145.79.227.12:4765 , piece_id=D4M33JNPTNF5EOTP6V4CH2KBZOJBKIRZ5RCEUMJMFSB5KEO3EITA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.528772708 (604h8m20.399855963s) [noise=true , address=77.90.31.21:4880 , piece_id=2EIJYMPCZP4BWQX4I3RSQS5IDOEAMX7UAGQIXEVRJQG6MPBZSFQA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.723971783 (604h8m20.595054818s) [noise=true , address=45.91.116.12:4227 , piece_id=Q26IH3AZJJISAEL5LPYT7CRFTP4KUMXOX44MYSG3BCLZNLMYLERQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.738010033 (604h8m20.609093068s) [noise=true , address=45.91.116.7:4012 , piece_id=LOW2MEEFBV64QEH7MKQT2R4PHUBPWPRZGDYVCYMELUD4BOQTGEVA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:20.900656063 (604h8m20.771739098s) [noise=true , address=45.91.116.8:4048 , piece_id=2MXNCFG5BZSBRLTW4OPJXJWAVDPMHK4GXYPWT6GBFIH65F3IEMVA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:21.200841094 (604h8m21.07192414s) [noise=true , address=77.90.31.12:4589 , piece_id=RQH5BFFWHPXUA4NMBZUJYNOAI5OQU43HDCGKJMYBSS6O3YDFQL2A, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:21.316516896 (604h8m21.187599941s) [noise=true , address=45.91.116.11:4118 , piece_id=UU2AYD5D3SDGOW7JMKI6CV2SNPTVZNKJUW4TFNWSQ22WH4LAOYTQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: <nil> Dec 30 19:37:21.316538046 (604h8m21.187621081s) [seg_pos={0 0}, upload=529518]: Segment reached optimal threshold of 13 pieces. Dec 30 19:37:21.316716626 (604h8m21.187799661s) [noise=true , address=145.79.227.29:4941 , piece_id=J2QTCA233FRN5E6KQXXIP767JHUL46VQJ7I76UTJEWVLO4UYAHFQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 30 19:37:21.316723229 (604h8m21.187806264s) [noise=true , address=145.79.227.29:4941 , piece_id=J2QTCA233FRN5E6KQXXIP767JHUL46VQJ7I76UTJEWVLO4UYAHFQ, seg_pos={0 0}, upload=529518]: Long tail cancellation due to optimal threshold Dec 30 19:37:21.316823179 (604h8m21.187906214s) [noise=true , address=45.91.116.13:4156 , piece_id=ZXMVGKUW2Q4SQNBGD5ZZP7CHF2RFVVLSZIYEMVOOZB2VU4CEU6UQ, seg_pos={0 0}, upload=529518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 30 19:37:21.316831525 (604h8m21.18791456s) [noise=true , address=45.91.116.13:4156 , piece_id=ZXMVGKUW2Q4SQNBGD5ZZP7CHF2RFVVLSZIYEMVOOZB2VU4CEU6UQ, seg_pos={0 0}, upload=529518]: Long tail cancellation due to optimal threshold Dec 30 19:37:21.317004414 (604h8m21.18808745s) [noise=true , address=145.79.227.23:4547 , piece_id=6G7N6QNXE3HHSWO6BMDRCQS26RJKV7PETS3VCAPLPCJNFHR4WGEA, seg_pos={0 0}, upload=529518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: write tcp 188.166.225.170:45998->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:45998->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45998->145.79.227.23:4547: use of closed network connection Dec 30 19:37:21.317013822 (604h8m21.188096877s) [noise=true , address=145.79.227.23:4547 , piece_id=6G7N6QNXE3HHSWO6BMDRCQS26RJKV7PETS3VCAPLPCJNFHR4WGEA, seg_pos={0 0}, upload=529518]: Long tail cancellation due to optimal threshold Dec 30 19:37:21.317033068 (604h8m21.188116113s) [seg_pos={0 0}, upload=529518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 19:37:21.317121227 (604h8m21.188204262s) [upload=529518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 19:37:21.317128490 (604h8m21.188211555s) [upload=529518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 19:37:21.616086595 (604h8m21.48716965s) [upload=529518]: Done uploading segments... Dec 30 19:37:21.616243293 (604h8m21.487326328s) [upload=529518]: Upload finished. err: <nil> Duration: 27.376206274s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)