Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.736596916s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:41:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 5.736632553s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.793739197s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:41:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 5.793770859s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 6.043804559s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:41:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 6.043832792s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 7.972598824s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:40:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 7.972626717s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 6.350539604s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:39:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 6.350567647s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.904173284s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:38:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 5.904200224s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.779417667s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:38:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 5.779444909s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.887372759s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:37:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 5.887399761s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.520067904s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:36:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 6.520098141s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 9.890316875s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:35:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 9.890344057s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.74600356s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:35:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 5.746029419s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.965430601s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:34:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 5.965458125s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.743443394s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:34:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 5.743486657s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.010108789s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:33:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 6.010124098s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.998390323s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:32:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 5.998419689s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.975500168s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:32:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 5.975533872s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.730192874s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:31:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 5.730218312s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.815512055s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:30:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 5.815537192s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.73474917s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:30:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 5.734779176s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.804717151s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:29:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 5.804735385s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.866393453s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:28:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 5.866420324s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.754025633s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:27:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 5.754052293s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.902559398s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:26:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 5.902637898s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.883713411s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:26:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 5.883743569s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.663971035s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:25:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 5.663993738s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.955715527s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:24:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 5.955745504s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.090188454s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:24:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 6.090214454s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.085069232s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:22:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 6.085093619s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.905136106s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:21:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 5.905157997s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.084881469s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:21:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 6.084911075s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.494496069s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:20:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 6.494517849s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.965057324s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:20:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 5.965084737s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 6.031972162s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:19:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 6.032015444s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.003947684s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:19:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 30.003972933s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.073038612s trace: https://jaeger.dev.storj.io/trace/672f829e95359b44 FAIL (2025-12-21 17:49:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, Piece ID: 2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ) context deadline exceeded; piecestore: (Node ID: 125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, Piece ID: XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ) context deadline exceeded; piecestore: (Node ID: 1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7, Piece ID: MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ) context deadline exceeded; piecestore: (Node ID: 1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR, Piece ID: TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA) context deadline exceeded; piecestore: (Node ID: 1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, Piece ID: NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA) context deadline exceeded; piecestore: (Node ID: 1uzGTks5tffchQBGQK23fnWZXMfrsTHh4Zqs9UGwnWUmnQghEW, Piece ID: DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ) context deadline exceeded ======LOGS====== :06:15.347427793 (385h37m15.218510828s) [upload=338446]: Starting upload Dec 21 17:06:15.347540547 (385h37m15.218623582s) [upload=338446]: Done starting upload Dec 21 17:06:15.347708697 (385h37m15.218791732s) [upload=338446]: Uploading segments... Dec 21 17:06:15.348224540 (385h37m15.219307585s) [upload=338446]: Got next segment. Inline: false Dec 21 17:06:15.348260819 (385h37m15.219343864s) [upload=338446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 17:06:15.348266620 (385h37m15.219349665s) [upload=338446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 17:06:15.594454120 (385h37m15.465537155s) [seg_pos={0 0}, upload=338446]: Begin upload segment... Dec 21 17:06:15.594522220 (385h37m15.465605265s) [seg_pos={0 0}, upload=338446]: Done begin upload segment. Dec 21 17:06:15.594559711 (385h37m15.465642756s) [noise=true , address=145.79.227.43:4157 , piece_id=WFODGDB7OPX5HU27AOYA2G4BGNDVBJTHC2AXHVJUWT7W5S3Y53QQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.594649401 (385h37m15.465732436s) [noise=true , address=145.79.227.52:4446 , piece_id=MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595436711 (385h37m15.466519746s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595545598 (385h37m15.466628663s) [noise=true , address=145.79.227.48:4302 , piece_id=TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595937244 (385h37m15.467020289s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596027827 (385h37m15.467110872s) [noise=true , address=145.79.227.15:4876 , piece_id=NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596436265 (385h37m15.46751931s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596452004 (385h37m15.467535049s) [noise=true , address=145.79.227.49:4337 , piece_id=QATOTFHA4NAMDIGFFPZ5HEEQCC4ENPK5N436NLJ7ZDTR4F3P5CZQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596871964 (385h37m15.467955009s) [noise=true , address=145.79.227.47:4266 , piece_id=DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596893134 (385h37m15.467976169s) [noise=true , address=145.79.227.46:4602 , piece_id=CKBC5AA4MCP6Q2BZV6ELAETITPAP2WDNC5KW4YQOPWM2EDFDYNYQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597290551 (385h37m15.468373596s) [noise=true , address=145.79.227.36:4517 , piece_id=IBBYGIBCO33ESOQSSVUHNRARK36RP5VGWGFLRX2TWT55CTHQIZVA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597380512 (385h37m15.468463547s) [noise=true , address=145.79.227.2:4115 , piece_id=2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597713376 (385h37m15.468796421s) [noise=true , address=145.79.227.45:4231 , piece_id=AIKXV2DOMBCG224JP6D7F4HCEAPLTI4KVNHOUZP5HJMU775HCAPA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597875485 (385h37m15.46895852s) [noise=true , address=145.79.227.38:4977 , piece_id=XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.598258684 (385h37m15.469341729s) [noise=true , address=145.79.227.11:4732 , piece_id=ZLEFYSAEDJSJN5JAB4CLPQ6HVPSU5G6C6QM4APGO6WO5LYCF5SJQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.598373042 (385h37m15.469456077s) [noise=true , address=145.79.227.9:4366 , piece_id=3W6BSQDQBHUN32FPT4BYXWXEJ5YB7PX5PFU425TKWX6WSMEZPE6A, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:17.445367236 (385h37m17.316450281s) [upload=338446]: Next returned nil segment Dec 21 17:06:17.445397093 (385h37m17.316480128s) [upload=338446]: Waiting for error group managing segments... Dec 21 17:06:17.757291870 (385h37m17.628374905s) [noise=true , address=145.79.227.46:4602 , piece_id=CKBC5AA4MCP6Q2BZV6ELAETITPAP2WDNC5KW4YQOPWM2EDFDYNYQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.761992040 (385h37m17.633075075s) [noise=true , address=145.79.227.43:4157 , piece_id=WFODGDB7OPX5HU27AOYA2G4BGNDVBJTHC2AXHVJUWT7W5S3Y53QQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.786066502 (385h37m17.657149547s) [noise=true , address=145.79.227.11:4732 , piece_id=ZLEFYSAEDJSJN5JAB4CLPQ6HVPSU5G6C6QM4APGO6WO5LYCF5SJQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.810862998 (385h37m17.681946043s) [noise=true , address=145.79.227.49:4337 , piece_id=QATOTFHA4NAMDIGFFPZ5HEEQCC4ENPK5N436NLJ7ZDTR4F3P5CZQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.828164223 (385h37m17.699247258s) [noise=true , address=145.79.227.9:4366 , piece_id=3W6BSQDQBHUN32FPT4BYXWXEJ5YB7PX5PFU425TKWX6WSMEZPE6A, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.830694551 (385h37m17.701777586s) [noise=true , address=145.79.227.15:4876 , piece_id=NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.853562054 (385h37m17.724645089s) [noise=true , address=145.79.227.2:4115 , piece_id=2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.856593877 (385h37m17.727676912s) [noise=true , address=145.79.227.52:4446 , piece_id=MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.168473827 (385h37m18.039556872s) [noise=true , address=145.79.227.45:4231 , piece_id=AIKXV2DOMBCG224JP6D7F4HCEAPLTI4KVNHOUZP5HJMU775HCAPA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.542043663 (385h37m18.413126698s) [noise=true , address=145.79.227.47:4266 , piece_id=DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.547438796 (385h37m18.418521831s) [noise=true , address=145.79.227.38:4977 , piece_id=XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.608531947 (385h37m18.479614982s) [noise=true , address=145.79.227.36:4517 , piece_id=IBBYGIBCO33ESOQSSVUHNRARK36RP5VGWGFLRX2TWT55CTHQIZVA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.827072654 (385h37m18.698155699s) [noise=true , address=145.79.227.48:4302 , piece_id=TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.827091900 (385h37m18.698174945s) [seg_pos={0 0}, upload=338446]: Segment reached optimal threshold of 13 pieces. Dec 21 17:06:18.827253068 (385h37m18.698336103s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection Dec 21 17:06:18.827314575 (385h37m18.69839764s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827579128 (385h37m18.698662173s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: expected piece hash; context canceled; EOF Dec 21 17:06:18.827585470 (385h37m18.698668505s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827702984 (385h37m18.698786019s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 21 17:06:18.827708485 (385h37m18.69879153s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827727400 (385h37m18.698810435s) [seg_pos={0 0}, upload=338446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 17:06:18.827789619 (385h37m18.698872654s) [upload=338446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 17:06:18.827794578 (385h37m18.698877613s) [upload=338446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 17:06:19.123495421 (385h37m18.994578456s) [upload=338446]: Done uploading segments... Dec 21 17:06:19.123517003 (385h37m18.994600048s) [upload=338446]: Upload finished. err: <nil> Duration: 6.073068761s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.730787019s trace: https://jaeger.dev.storj.io/trace/672f829e95359b44 FAIL (2025-12-21 17:49:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, Piece ID: 2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ) context deadline exceeded; piecestore: (Node ID: 125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, Piece ID: XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ) context deadline exceeded; piecestore: (Node ID: 1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7, Piece ID: MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ) context deadline exceeded; piecestore: (Node ID: 1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR, Piece ID: TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA) context deadline exceeded; piecestore: (Node ID: 1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, Piece ID: NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA) context deadline exceeded; piecestore: (Node ID: 1uzGTks5tffchQBGQK23fnWZXMfrsTHh4Zqs9UGwnWUmnQghEW, Piece ID: DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ) context deadline exceeded ======LOGS====== :06:15.347427793 (385h37m15.218510828s) [upload=338446]: Starting upload Dec 21 17:06:15.347540547 (385h37m15.218623582s) [upload=338446]: Done starting upload Dec 21 17:06:15.347708697 (385h37m15.218791732s) [upload=338446]: Uploading segments... Dec 21 17:06:15.348224540 (385h37m15.219307585s) [upload=338446]: Got next segment. Inline: false Dec 21 17:06:15.348260819 (385h37m15.219343864s) [upload=338446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 17:06:15.348266620 (385h37m15.219349665s) [upload=338446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 17:06:15.594454120 (385h37m15.465537155s) [seg_pos={0 0}, upload=338446]: Begin upload segment... Dec 21 17:06:15.594522220 (385h37m15.465605265s) [seg_pos={0 0}, upload=338446]: Done begin upload segment. Dec 21 17:06:15.594559711 (385h37m15.465642756s) [noise=true , address=145.79.227.43:4157 , piece_id=WFODGDB7OPX5HU27AOYA2G4BGNDVBJTHC2AXHVJUWT7W5S3Y53QQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.594649401 (385h37m15.465732436s) [noise=true , address=145.79.227.52:4446 , piece_id=MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595436711 (385h37m15.466519746s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595545598 (385h37m15.466628663s) [noise=true , address=145.79.227.48:4302 , piece_id=TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595937244 (385h37m15.467020289s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596027827 (385h37m15.467110872s) [noise=true , address=145.79.227.15:4876 , piece_id=NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596436265 (385h37m15.46751931s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596452004 (385h37m15.467535049s) [noise=true , address=145.79.227.49:4337 , piece_id=QATOTFHA4NAMDIGFFPZ5HEEQCC4ENPK5N436NLJ7ZDTR4F3P5CZQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596871964 (385h37m15.467955009s) [noise=true , address=145.79.227.47:4266 , piece_id=DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596893134 (385h37m15.467976169s) [noise=true , address=145.79.227.46:4602 , piece_id=CKBC5AA4MCP6Q2BZV6ELAETITPAP2WDNC5KW4YQOPWM2EDFDYNYQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597290551 (385h37m15.468373596s) [noise=true , address=145.79.227.36:4517 , piece_id=IBBYGIBCO33ESOQSSVUHNRARK36RP5VGWGFLRX2TWT55CTHQIZVA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597380512 (385h37m15.468463547s) [noise=true , address=145.79.227.2:4115 , piece_id=2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597713376 (385h37m15.468796421s) [noise=true , address=145.79.227.45:4231 , piece_id=AIKXV2DOMBCG224JP6D7F4HCEAPLTI4KVNHOUZP5HJMU775HCAPA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597875485 (385h37m15.46895852s) [noise=true , address=145.79.227.38:4977 , piece_id=XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.598258684 (385h37m15.469341729s) [noise=true , address=145.79.227.11:4732 , piece_id=ZLEFYSAEDJSJN5JAB4CLPQ6HVPSU5G6C6QM4APGO6WO5LYCF5SJQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.598373042 (385h37m15.469456077s) [noise=true , address=145.79.227.9:4366 , piece_id=3W6BSQDQBHUN32FPT4BYXWXEJ5YB7PX5PFU425TKWX6WSMEZPE6A, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:17.445367236 (385h37m17.316450281s) [upload=338446]: Next returned nil segment Dec 21 17:06:17.445397093 (385h37m17.316480128s) [upload=338446]: Waiting for error group managing segments... Dec 21 17:06:17.757291870 (385h37m17.628374905s) [noise=true , address=145.79.227.46:4602 , piece_id=CKBC5AA4MCP6Q2BZV6ELAETITPAP2WDNC5KW4YQOPWM2EDFDYNYQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.761992040 (385h37m17.633075075s) [noise=true , address=145.79.227.43:4157 , piece_id=WFODGDB7OPX5HU27AOYA2G4BGNDVBJTHC2AXHVJUWT7W5S3Y53QQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.786066502 (385h37m17.657149547s) [noise=true , address=145.79.227.11:4732 , piece_id=ZLEFYSAEDJSJN5JAB4CLPQ6HVPSU5G6C6QM4APGO6WO5LYCF5SJQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.810862998 (385h37m17.681946043s) [noise=true , address=145.79.227.49:4337 , piece_id=QATOTFHA4NAMDIGFFPZ5HEEQCC4ENPK5N436NLJ7ZDTR4F3P5CZQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.828164223 (385h37m17.699247258s) [noise=true , address=145.79.227.9:4366 , piece_id=3W6BSQDQBHUN32FPT4BYXWXEJ5YB7PX5PFU425TKWX6WSMEZPE6A, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.830694551 (385h37m17.701777586s) [noise=true , address=145.79.227.15:4876 , piece_id=NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.853562054 (385h37m17.724645089s) [noise=true , address=145.79.227.2:4115 , piece_id=2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.856593877 (385h37m17.727676912s) [noise=true , address=145.79.227.52:4446 , piece_id=MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.168473827 (385h37m18.039556872s) [noise=true , address=145.79.227.45:4231 , piece_id=AIKXV2DOMBCG224JP6D7F4HCEAPLTI4KVNHOUZP5HJMU775HCAPA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.542043663 (385h37m18.413126698s) [noise=true , address=145.79.227.47:4266 , piece_id=DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.547438796 (385h37m18.418521831s) [noise=true , address=145.79.227.38:4977 , piece_id=XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.608531947 (385h37m18.479614982s) [noise=true , address=145.79.227.36:4517 , piece_id=IBBYGIBCO33ESOQSSVUHNRARK36RP5VGWGFLRX2TWT55CTHQIZVA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.827072654 (385h37m18.698155699s) [noise=true , address=145.79.227.48:4302 , piece_id=TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.827091900 (385h37m18.698174945s) [seg_pos={0 0}, upload=338446]: Segment reached optimal threshold of 13 pieces. Dec 21 17:06:18.827253068 (385h37m18.698336103s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection Dec 21 17:06:18.827314575 (385h37m18.69839764s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827579128 (385h37m18.698662173s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: expected piece hash; context canceled; EOF Dec 21 17:06:18.827585470 (385h37m18.698668505s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827702984 (385h37m18.698786019s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 21 17:06:18.827708485 (385h37m18.69879153s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827727400 (385h37m18.698810435s) [seg_pos={0 0}, upload=338446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 17:06:18.827789619 (385h37m18.698872654s) [upload=338446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 17:06:18.827794578 (385h37m18.698877613s) [upload=338446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 17:06:19.123495421 (385h37m18.994578456s) [upload=338446]: Done uploading segments... Dec 21 17:06:19.123517003 (385h37m18.994600048s) [upload=338446]: Upload finished. err: <nil> Duration: 5.730885356s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.X.........................................................] duration: 5.93323191s trace: https://jaeger.dev.storj.io/trace/672f829e95359b44 FAIL (2025-12-21 17:48:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, Piece ID: 2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ) context deadline exceeded; piecestore: (Node ID: 125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, Piece ID: XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ) context deadline exceeded; piecestore: (Node ID: 1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7, Piece ID: MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ) context deadline exceeded; piecestore: (Node ID: 1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR, Piece ID: TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA) context deadline exceeded; piecestore: (Node ID: 1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, Piece ID: NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA) context deadline exceeded; piecestore: (Node ID: 1uzGTks5tffchQBGQK23fnWZXMfrsTHh4Zqs9UGwnWUmnQghEW, Piece ID: DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ) context deadline exceeded ======LOGS====== :06:15.347427793 (385h37m15.218510828s) [upload=338446]: Starting upload Dec 21 17:06:15.347540547 (385h37m15.218623582s) [upload=338446]: Done starting upload Dec 21 17:06:15.347708697 (385h37m15.218791732s) [upload=338446]: Uploading segments... Dec 21 17:06:15.348224540 (385h37m15.219307585s) [upload=338446]: Got next segment. Inline: false Dec 21 17:06:15.348260819 (385h37m15.219343864s) [upload=338446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 17:06:15.348266620 (385h37m15.219349665s) [upload=338446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 17:06:15.594454120 (385h37m15.465537155s) [seg_pos={0 0}, upload=338446]: Begin upload segment... Dec 21 17:06:15.594522220 (385h37m15.465605265s) [seg_pos={0 0}, upload=338446]: Done begin upload segment. Dec 21 17:06:15.594559711 (385h37m15.465642756s) [noise=true , address=145.79.227.43:4157 , piece_id=WFODGDB7OPX5HU27AOYA2G4BGNDVBJTHC2AXHVJUWT7W5S3Y53QQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.594649401 (385h37m15.465732436s) [noise=true , address=145.79.227.52:4446 , piece_id=MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595436711 (385h37m15.466519746s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595545598 (385h37m15.466628663s) [noise=true , address=145.79.227.48:4302 , piece_id=TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595937244 (385h37m15.467020289s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596027827 (385h37m15.467110872s) [noise=true , address=145.79.227.15:4876 , piece_id=NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596436265 (385h37m15.46751931s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596452004 (385h37m15.467535049s) [noise=true , address=145.79.227.49:4337 , piece_id=QATOTFHA4NAMDIGFFPZ5HEEQCC4ENPK5N436NLJ7ZDTR4F3P5CZQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596871964 (385h37m15.467955009s) [noise=true , address=145.79.227.47:4266 , piece_id=DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596893134 (385h37m15.467976169s) [noise=true , address=145.79.227.46:4602 , piece_id=CKBC5AA4MCP6Q2BZV6ELAETITPAP2WDNC5KW4YQOPWM2EDFDYNYQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597290551 (385h37m15.468373596s) [noise=true , address=145.79.227.36:4517 , piece_id=IBBYGIBCO33ESOQSSVUHNRARK36RP5VGWGFLRX2TWT55CTHQIZVA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597380512 (385h37m15.468463547s) [noise=true , address=145.79.227.2:4115 , piece_id=2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597713376 (385h37m15.468796421s) [noise=true , address=145.79.227.45:4231 , piece_id=AIKXV2DOMBCG224JP6D7F4HCEAPLTI4KVNHOUZP5HJMU775HCAPA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597875485 (385h37m15.46895852s) [noise=true , address=145.79.227.38:4977 , piece_id=XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.598258684 (385h37m15.469341729s) [noise=true , address=145.79.227.11:4732 , piece_id=ZLEFYSAEDJSJN5JAB4CLPQ6HVPSU5G6C6QM4APGO6WO5LYCF5SJQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.598373042 (385h37m15.469456077s) [noise=true , address=145.79.227.9:4366 , piece_id=3W6BSQDQBHUN32FPT4BYXWXEJ5YB7PX5PFU425TKWX6WSMEZPE6A, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:17.445367236 (385h37m17.316450281s) [upload=338446]: Next returned nil segment Dec 21 17:06:17.445397093 (385h37m17.316480128s) [upload=338446]: Waiting for error group managing segments... Dec 21 17:06:17.757291870 (385h37m17.628374905s) [noise=true , address=145.79.227.46:4602 , piece_id=CKBC5AA4MCP6Q2BZV6ELAETITPAP2WDNC5KW4YQOPWM2EDFDYNYQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.761992040 (385h37m17.633075075s) [noise=true , address=145.79.227.43:4157 , piece_id=WFODGDB7OPX5HU27AOYA2G4BGNDVBJTHC2AXHVJUWT7W5S3Y53QQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.786066502 (385h37m17.657149547s) [noise=true , address=145.79.227.11:4732 , piece_id=ZLEFYSAEDJSJN5JAB4CLPQ6HVPSU5G6C6QM4APGO6WO5LYCF5SJQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.810862998 (385h37m17.681946043s) [noise=true , address=145.79.227.49:4337 , piece_id=QATOTFHA4NAMDIGFFPZ5HEEQCC4ENPK5N436NLJ7ZDTR4F3P5CZQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.828164223 (385h37m17.699247258s) [noise=true , address=145.79.227.9:4366 , piece_id=3W6BSQDQBHUN32FPT4BYXWXEJ5YB7PX5PFU425TKWX6WSMEZPE6A, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.830694551 (385h37m17.701777586s) [noise=true , address=145.79.227.15:4876 , piece_id=NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.853562054 (385h37m17.724645089s) [noise=true , address=145.79.227.2:4115 , piece_id=2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.856593877 (385h37m17.727676912s) [noise=true , address=145.79.227.52:4446 , piece_id=MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.168473827 (385h37m18.039556872s) [noise=true , address=145.79.227.45:4231 , piece_id=AIKXV2DOMBCG224JP6D7F4HCEAPLTI4KVNHOUZP5HJMU775HCAPA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.542043663 (385h37m18.413126698s) [noise=true , address=145.79.227.47:4266 , piece_id=DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.547438796 (385h37m18.418521831s) [noise=true , address=145.79.227.38:4977 , piece_id=XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.608531947 (385h37m18.479614982s) [noise=true , address=145.79.227.36:4517 , piece_id=IBBYGIBCO33ESOQSSVUHNRARK36RP5VGWGFLRX2TWT55CTHQIZVA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.827072654 (385h37m18.698155699s) [noise=true , address=145.79.227.48:4302 , piece_id=TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.827091900 (385h37m18.698174945s) [seg_pos={0 0}, upload=338446]: Segment reached optimal threshold of 13 pieces. Dec 21 17:06:18.827253068 (385h37m18.698336103s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection Dec 21 17:06:18.827314575 (385h37m18.69839764s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827579128 (385h37m18.698662173s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: expected piece hash; context canceled; EOF Dec 21 17:06:18.827585470 (385h37m18.698668505s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827702984 (385h37m18.698786019s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 21 17:06:18.827708485 (385h37m18.69879153s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827727400 (385h37m18.698810435s) [seg_pos={0 0}, upload=338446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 17:06:18.827789619 (385h37m18.698872654s) [upload=338446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 17:06:18.827794578 (385h37m18.698877613s) [upload=338446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 17:06:19.123495421 (385h37m18.994578456s) [upload=338446]: Done uploading segments... Dec 21 17:06:19.123517003 (385h37m18.994600048s) [upload=338446]: Upload finished. err: <nil> Duration: 5.933267938s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X.X........................................................] duration: 6.339152233s trace: https://jaeger.dev.storj.io/trace/672f829e95359b44 FAIL (2025-12-21 17:48:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, Piece ID: 2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ) context deadline exceeded; piecestore: (Node ID: 125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, Piece ID: XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ) context deadline exceeded; piecestore: (Node ID: 1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7, Piece ID: MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ) context deadline exceeded; piecestore: (Node ID: 1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR, Piece ID: TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA) context deadline exceeded; piecestore: (Node ID: 1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, Piece ID: NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA) context deadline exceeded; piecestore: (Node ID: 1uzGTks5tffchQBGQK23fnWZXMfrsTHh4Zqs9UGwnWUmnQghEW, Piece ID: DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ) context deadline exceeded ======LOGS====== :06:15.347427793 (385h37m15.218510828s) [upload=338446]: Starting upload Dec 21 17:06:15.347540547 (385h37m15.218623582s) [upload=338446]: Done starting upload Dec 21 17:06:15.347708697 (385h37m15.218791732s) [upload=338446]: Uploading segments... Dec 21 17:06:15.348224540 (385h37m15.219307585s) [upload=338446]: Got next segment. Inline: false Dec 21 17:06:15.348260819 (385h37m15.219343864s) [upload=338446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 17:06:15.348266620 (385h37m15.219349665s) [upload=338446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 17:06:15.594454120 (385h37m15.465537155s) [seg_pos={0 0}, upload=338446]: Begin upload segment... Dec 21 17:06:15.594522220 (385h37m15.465605265s) [seg_pos={0 0}, upload=338446]: Done begin upload segment. Dec 21 17:06:15.594559711 (385h37m15.465642756s) [noise=true , address=145.79.227.43:4157 , piece_id=WFODGDB7OPX5HU27AOYA2G4BGNDVBJTHC2AXHVJUWT7W5S3Y53QQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.594649401 (385h37m15.465732436s) [noise=true , address=145.79.227.52:4446 , piece_id=MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595436711 (385h37m15.466519746s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595545598 (385h37m15.466628663s) [noise=true , address=145.79.227.48:4302 , piece_id=TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595937244 (385h37m15.467020289s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596027827 (385h37m15.467110872s) [noise=true , address=145.79.227.15:4876 , piece_id=NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596436265 (385h37m15.46751931s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596452004 (385h37m15.467535049s) [noise=true , address=145.79.227.49:4337 , piece_id=QATOTFHA4NAMDIGFFPZ5HEEQCC4ENPK5N436NLJ7ZDTR4F3P5CZQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596871964 (385h37m15.467955009s) [noise=true , address=145.79.227.47:4266 , piece_id=DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596893134 (385h37m15.467976169s) [noise=true , address=145.79.227.46:4602 , piece_id=CKBC5AA4MCP6Q2BZV6ELAETITPAP2WDNC5KW4YQOPWM2EDFDYNYQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597290551 (385h37m15.468373596s) [noise=true , address=145.79.227.36:4517 , piece_id=IBBYGIBCO33ESOQSSVUHNRARK36RP5VGWGFLRX2TWT55CTHQIZVA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597380512 (385h37m15.468463547s) [noise=true , address=145.79.227.2:4115 , piece_id=2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597713376 (385h37m15.468796421s) [noise=true , address=145.79.227.45:4231 , piece_id=AIKXV2DOMBCG224JP6D7F4HCEAPLTI4KVNHOUZP5HJMU775HCAPA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597875485 (385h37m15.46895852s) [noise=true , address=145.79.227.38:4977 , piece_id=XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.598258684 (385h37m15.469341729s) [noise=true , address=145.79.227.11:4732 , piece_id=ZLEFYSAEDJSJN5JAB4CLPQ6HVPSU5G6C6QM4APGO6WO5LYCF5SJQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.598373042 (385h37m15.469456077s) [noise=true , address=145.79.227.9:4366 , piece_id=3W6BSQDQBHUN32FPT4BYXWXEJ5YB7PX5PFU425TKWX6WSMEZPE6A, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:17.445367236 (385h37m17.316450281s) [upload=338446]: Next returned nil segment Dec 21 17:06:17.445397093 (385h37m17.316480128s) [upload=338446]: Waiting for error group managing segments... Dec 21 17:06:17.757291870 (385h37m17.628374905s) [noise=true , address=145.79.227.46:4602 , piece_id=CKBC5AA4MCP6Q2BZV6ELAETITPAP2WDNC5KW4YQOPWM2EDFDYNYQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.761992040 (385h37m17.633075075s) [noise=true , address=145.79.227.43:4157 , piece_id=WFODGDB7OPX5HU27AOYA2G4BGNDVBJTHC2AXHVJUWT7W5S3Y53QQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.786066502 (385h37m17.657149547s) [noise=true , address=145.79.227.11:4732 , piece_id=ZLEFYSAEDJSJN5JAB4CLPQ6HVPSU5G6C6QM4APGO6WO5LYCF5SJQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.810862998 (385h37m17.681946043s) [noise=true , address=145.79.227.49:4337 , piece_id=QATOTFHA4NAMDIGFFPZ5HEEQCC4ENPK5N436NLJ7ZDTR4F3P5CZQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.828164223 (385h37m17.699247258s) [noise=true , address=145.79.227.9:4366 , piece_id=3W6BSQDQBHUN32FPT4BYXWXEJ5YB7PX5PFU425TKWX6WSMEZPE6A, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.830694551 (385h37m17.701777586s) [noise=true , address=145.79.227.15:4876 , piece_id=NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.853562054 (385h37m17.724645089s) [noise=true , address=145.79.227.2:4115 , piece_id=2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.856593877 (385h37m17.727676912s) [noise=true , address=145.79.227.52:4446 , piece_id=MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.168473827 (385h37m18.039556872s) [noise=true , address=145.79.227.45:4231 , piece_id=AIKXV2DOMBCG224JP6D7F4HCEAPLTI4KVNHOUZP5HJMU775HCAPA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.542043663 (385h37m18.413126698s) [noise=true , address=145.79.227.47:4266 , piece_id=DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.547438796 (385h37m18.418521831s) [noise=true , address=145.79.227.38:4977 , piece_id=XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.608531947 (385h37m18.479614982s) [noise=true , address=145.79.227.36:4517 , piece_id=IBBYGIBCO33ESOQSSVUHNRARK36RP5VGWGFLRX2TWT55CTHQIZVA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.827072654 (385h37m18.698155699s) [noise=true , address=145.79.227.48:4302 , piece_id=TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.827091900 (385h37m18.698174945s) [seg_pos={0 0}, upload=338446]: Segment reached optimal threshold of 13 pieces. Dec 21 17:06:18.827253068 (385h37m18.698336103s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection Dec 21 17:06:18.827314575 (385h37m18.69839764s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827579128 (385h37m18.698662173s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: expected piece hash; context canceled; EOF Dec 21 17:06:18.827585470 (385h37m18.698668505s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827702984 (385h37m18.698786019s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 21 17:06:18.827708485 (385h37m18.69879153s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827727400 (385h37m18.698810435s) [seg_pos={0 0}, upload=338446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 17:06:18.827789619 (385h37m18.698872654s) [upload=338446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 17:06:18.827794578 (385h37m18.698877613s) [upload=338446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 17:06:19.123495421 (385h37m18.994578456s) [upload=338446]: Done uploading segments... Dec 21 17:06:19.123517003 (385h37m18.994600048s) [upload=338446]: Upload finished. err: <nil> Duration: 6.33917695s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X.X.X.......................................................] duration: 5.991696464s trace: https://jaeger.dev.storj.io/trace/672f829e95359b44 FAIL (2025-12-21 17:47:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, Piece ID: 2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ) context deadline exceeded; piecestore: (Node ID: 125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, Piece ID: XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ) context deadline exceeded; piecestore: (Node ID: 1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7, Piece ID: MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ) context deadline exceeded; piecestore: (Node ID: 1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR, Piece ID: TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA) context deadline exceeded; piecestore: (Node ID: 1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, Piece ID: NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA) context deadline exceeded; piecestore: (Node ID: 1uzGTks5tffchQBGQK23fnWZXMfrsTHh4Zqs9UGwnWUmnQghEW, Piece ID: DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ) context deadline exceeded ======LOGS====== :06:15.347427793 (385h37m15.218510828s) [upload=338446]: Starting upload Dec 21 17:06:15.347540547 (385h37m15.218623582s) [upload=338446]: Done starting upload Dec 21 17:06:15.347708697 (385h37m15.218791732s) [upload=338446]: Uploading segments... Dec 21 17:06:15.348224540 (385h37m15.219307585s) [upload=338446]: Got next segment. Inline: false Dec 21 17:06:15.348260819 (385h37m15.219343864s) [upload=338446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 17:06:15.348266620 (385h37m15.219349665s) [upload=338446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 17:06:15.594454120 (385h37m15.465537155s) [seg_pos={0 0}, upload=338446]: Begin upload segment... Dec 21 17:06:15.594522220 (385h37m15.465605265s) [seg_pos={0 0}, upload=338446]: Done begin upload segment. Dec 21 17:06:15.594559711 (385h37m15.465642756s) [noise=true , address=145.79.227.43:4157 , piece_id=WFODGDB7OPX5HU27AOYA2G4BGNDVBJTHC2AXHVJUWT7W5S3Y53QQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.594649401 (385h37m15.465732436s) [noise=true , address=145.79.227.52:4446 , piece_id=MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595436711 (385h37m15.466519746s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595545598 (385h37m15.466628663s) [noise=true , address=145.79.227.48:4302 , piece_id=TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595937244 (385h37m15.467020289s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596027827 (385h37m15.467110872s) [noise=true , address=145.79.227.15:4876 , piece_id=NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596436265 (385h37m15.46751931s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596452004 (385h37m15.467535049s) [noise=true , address=145.79.227.49:4337 , piece_id=QATOTFHA4NAMDIGFFPZ5HEEQCC4ENPK5N436NLJ7ZDTR4F3P5CZQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596871964 (385h37m15.467955009s) [noise=true , address=145.79.227.47:4266 , piece_id=DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596893134 (385h37m15.467976169s) [noise=true , address=145.79.227.46:4602 , piece_id=CKBC5AA4MCP6Q2BZV6ELAETITPAP2WDNC5KW4YQOPWM2EDFDYNYQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597290551 (385h37m15.468373596s) [noise=true , address=145.79.227.36:4517 , piece_id=IBBYGIBCO33ESOQSSVUHNRARK36RP5VGWGFLRX2TWT55CTHQIZVA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597380512 (385h37m15.468463547s) [noise=true , address=145.79.227.2:4115 , piece_id=2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597713376 (385h37m15.468796421s) [noise=true , address=145.79.227.45:4231 , piece_id=AIKXV2DOMBCG224JP6D7F4HCEAPLTI4KVNHOUZP5HJMU775HCAPA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597875485 (385h37m15.46895852s) [noise=true , address=145.79.227.38:4977 , piece_id=XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.598258684 (385h37m15.469341729s) [noise=true , address=145.79.227.11:4732 , piece_id=ZLEFYSAEDJSJN5JAB4CLPQ6HVPSU5G6C6QM4APGO6WO5LYCF5SJQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.598373042 (385h37m15.469456077s) [noise=true , address=145.79.227.9:4366 , piece_id=3W6BSQDQBHUN32FPT4BYXWXEJ5YB7PX5PFU425TKWX6WSMEZPE6A, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:17.445367236 (385h37m17.316450281s) [upload=338446]: Next returned nil segment Dec 21 17:06:17.445397093 (385h37m17.316480128s) [upload=338446]: Waiting for error group managing segments... Dec 21 17:06:17.757291870 (385h37m17.628374905s) [noise=true , address=145.79.227.46:4602 , piece_id=CKBC5AA4MCP6Q2BZV6ELAETITPAP2WDNC5KW4YQOPWM2EDFDYNYQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.761992040 (385h37m17.633075075s) [noise=true , address=145.79.227.43:4157 , piece_id=WFODGDB7OPX5HU27AOYA2G4BGNDVBJTHC2AXHVJUWT7W5S3Y53QQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.786066502 (385h37m17.657149547s) [noise=true , address=145.79.227.11:4732 , piece_id=ZLEFYSAEDJSJN5JAB4CLPQ6HVPSU5G6C6QM4APGO6WO5LYCF5SJQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.810862998 (385h37m17.681946043s) [noise=true , address=145.79.227.49:4337 , piece_id=QATOTFHA4NAMDIGFFPZ5HEEQCC4ENPK5N436NLJ7ZDTR4F3P5CZQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.828164223 (385h37m17.699247258s) [noise=true , address=145.79.227.9:4366 , piece_id=3W6BSQDQBHUN32FPT4BYXWXEJ5YB7PX5PFU425TKWX6WSMEZPE6A, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.830694551 (385h37m17.701777586s) [noise=true , address=145.79.227.15:4876 , piece_id=NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.853562054 (385h37m17.724645089s) [noise=true , address=145.79.227.2:4115 , piece_id=2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.856593877 (385h37m17.727676912s) [noise=true , address=145.79.227.52:4446 , piece_id=MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.168473827 (385h37m18.039556872s) [noise=true , address=145.79.227.45:4231 , piece_id=AIKXV2DOMBCG224JP6D7F4HCEAPLTI4KVNHOUZP5HJMU775HCAPA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.542043663 (385h37m18.413126698s) [noise=true , address=145.79.227.47:4266 , piece_id=DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.547438796 (385h37m18.418521831s) [noise=true , address=145.79.227.38:4977 , piece_id=XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.608531947 (385h37m18.479614982s) [noise=true , address=145.79.227.36:4517 , piece_id=IBBYGIBCO33ESOQSSVUHNRARK36RP5VGWGFLRX2TWT55CTHQIZVA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.827072654 (385h37m18.698155699s) [noise=true , address=145.79.227.48:4302 , piece_id=TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.827091900 (385h37m18.698174945s) [seg_pos={0 0}, upload=338446]: Segment reached optimal threshold of 13 pieces. Dec 21 17:06:18.827253068 (385h37m18.698336103s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection Dec 21 17:06:18.827314575 (385h37m18.69839764s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827579128 (385h37m18.698662173s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: expected piece hash; context canceled; EOF Dec 21 17:06:18.827585470 (385h37m18.698668505s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827702984 (385h37m18.698786019s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 21 17:06:18.827708485 (385h37m18.69879153s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827727400 (385h37m18.698810435s) [seg_pos={0 0}, upload=338446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 17:06:18.827789619 (385h37m18.698872654s) [upload=338446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 17:06:18.827794578 (385h37m18.698877613s) [upload=338446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 17:06:19.123495421 (385h37m18.994578456s) [upload=338446]: Done uploading segments... Dec 21 17:06:19.123517003 (385h37m18.994600048s) [upload=338446]: Upload finished. err: <nil> Duration: 5.991724167s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X.X.X......................................................] duration: 6.211959523s trace: https://jaeger.dev.storj.io/trace/672f829e95359b44 FAIL (2025-12-21 17:46:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, Piece ID: 2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ) context deadline exceeded; piecestore: (Node ID: 125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, Piece ID: XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ) context deadline exceeded; piecestore: (Node ID: 1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7, Piece ID: MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ) context deadline exceeded; piecestore: (Node ID: 1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR, Piece ID: TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA) context deadline exceeded; piecestore: (Node ID: 1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, Piece ID: NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA) context deadline exceeded; piecestore: (Node ID: 1uzGTks5tffchQBGQK23fnWZXMfrsTHh4Zqs9UGwnWUmnQghEW, Piece ID: DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ) context deadline exceeded ======LOGS====== :06:15.347427793 (385h37m15.218510828s) [upload=338446]: Starting upload Dec 21 17:06:15.347540547 (385h37m15.218623582s) [upload=338446]: Done starting upload Dec 21 17:06:15.347708697 (385h37m15.218791732s) [upload=338446]: Uploading segments... Dec 21 17:06:15.348224540 (385h37m15.219307585s) [upload=338446]: Got next segment. Inline: false Dec 21 17:06:15.348260819 (385h37m15.219343864s) [upload=338446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 17:06:15.348266620 (385h37m15.219349665s) [upload=338446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 17:06:15.594454120 (385h37m15.465537155s) [seg_pos={0 0}, upload=338446]: Begin upload segment... Dec 21 17:06:15.594522220 (385h37m15.465605265s) [seg_pos={0 0}, upload=338446]: Done begin upload segment. Dec 21 17:06:15.594559711 (385h37m15.465642756s) [noise=true , address=145.79.227.43:4157 , piece_id=WFODGDB7OPX5HU27AOYA2G4BGNDVBJTHC2AXHVJUWT7W5S3Y53QQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.594649401 (385h37m15.465732436s) [noise=true , address=145.79.227.52:4446 , piece_id=MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595436711 (385h37m15.466519746s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595545598 (385h37m15.466628663s) [noise=true , address=145.79.227.48:4302 , piece_id=TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595937244 (385h37m15.467020289s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596027827 (385h37m15.467110872s) [noise=true , address=145.79.227.15:4876 , piece_id=NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596436265 (385h37m15.46751931s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596452004 (385h37m15.467535049s) [noise=true , address=145.79.227.49:4337 , piece_id=QATOTFHA4NAMDIGFFPZ5HEEQCC4ENPK5N436NLJ7ZDTR4F3P5CZQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596871964 (385h37m15.467955009s) [noise=true , address=145.79.227.47:4266 , piece_id=DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596893134 (385h37m15.467976169s) [noise=true , address=145.79.227.46:4602 , piece_id=CKBC5AA4MCP6Q2BZV6ELAETITPAP2WDNC5KW4YQOPWM2EDFDYNYQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597290551 (385h37m15.468373596s) [noise=true , address=145.79.227.36:4517 , piece_id=IBBYGIBCO33ESOQSSVUHNRARK36RP5VGWGFLRX2TWT55CTHQIZVA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597380512 (385h37m15.468463547s) [noise=true , address=145.79.227.2:4115 , piece_id=2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597713376 (385h37m15.468796421s) [noise=true , address=145.79.227.45:4231 , piece_id=AIKXV2DOMBCG224JP6D7F4HCEAPLTI4KVNHOUZP5HJMU775HCAPA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597875485 (385h37m15.46895852s) [noise=true , address=145.79.227.38:4977 , piece_id=XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.598258684 (385h37m15.469341729s) [noise=true , address=145.79.227.11:4732 , piece_id=ZLEFYSAEDJSJN5JAB4CLPQ6HVPSU5G6C6QM4APGO6WO5LYCF5SJQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.598373042 (385h37m15.469456077s) [noise=true , address=145.79.227.9:4366 , piece_id=3W6BSQDQBHUN32FPT4BYXWXEJ5YB7PX5PFU425TKWX6WSMEZPE6A, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:17.445367236 (385h37m17.316450281s) [upload=338446]: Next returned nil segment Dec 21 17:06:17.445397093 (385h37m17.316480128s) [upload=338446]: Waiting for error group managing segments... Dec 21 17:06:17.757291870 (385h37m17.628374905s) [noise=true , address=145.79.227.46:4602 , piece_id=CKBC5AA4MCP6Q2BZV6ELAETITPAP2WDNC5KW4YQOPWM2EDFDYNYQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.761992040 (385h37m17.633075075s) [noise=true , address=145.79.227.43:4157 , piece_id=WFODGDB7OPX5HU27AOYA2G4BGNDVBJTHC2AXHVJUWT7W5S3Y53QQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.786066502 (385h37m17.657149547s) [noise=true , address=145.79.227.11:4732 , piece_id=ZLEFYSAEDJSJN5JAB4CLPQ6HVPSU5G6C6QM4APGO6WO5LYCF5SJQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.810862998 (385h37m17.681946043s) [noise=true , address=145.79.227.49:4337 , piece_id=QATOTFHA4NAMDIGFFPZ5HEEQCC4ENPK5N436NLJ7ZDTR4F3P5CZQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.828164223 (385h37m17.699247258s) [noise=true , address=145.79.227.9:4366 , piece_id=3W6BSQDQBHUN32FPT4BYXWXEJ5YB7PX5PFU425TKWX6WSMEZPE6A, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.830694551 (385h37m17.701777586s) [noise=true , address=145.79.227.15:4876 , piece_id=NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.853562054 (385h37m17.724645089s) [noise=true , address=145.79.227.2:4115 , piece_id=2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.856593877 (385h37m17.727676912s) [noise=true , address=145.79.227.52:4446 , piece_id=MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.168473827 (385h37m18.039556872s) [noise=true , address=145.79.227.45:4231 , piece_id=AIKXV2DOMBCG224JP6D7F4HCEAPLTI4KVNHOUZP5HJMU775HCAPA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.542043663 (385h37m18.413126698s) [noise=true , address=145.79.227.47:4266 , piece_id=DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.547438796 (385h37m18.418521831s) [noise=true , address=145.79.227.38:4977 , piece_id=XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.608531947 (385h37m18.479614982s) [noise=true , address=145.79.227.36:4517 , piece_id=IBBYGIBCO33ESOQSSVUHNRARK36RP5VGWGFLRX2TWT55CTHQIZVA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.827072654 (385h37m18.698155699s) [noise=true , address=145.79.227.48:4302 , piece_id=TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.827091900 (385h37m18.698174945s) [seg_pos={0 0}, upload=338446]: Segment reached optimal threshold of 13 pieces. Dec 21 17:06:18.827253068 (385h37m18.698336103s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection Dec 21 17:06:18.827314575 (385h37m18.69839764s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827579128 (385h37m18.698662173s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: expected piece hash; context canceled; EOF Dec 21 17:06:18.827585470 (385h37m18.698668505s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827702984 (385h37m18.698786019s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 21 17:06:18.827708485 (385h37m18.69879153s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827727400 (385h37m18.698810435s) [seg_pos={0 0}, upload=338446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 17:06:18.827789619 (385h37m18.698872654s) [upload=338446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 17:06:18.827794578 (385h37m18.698877613s) [upload=338446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 17:06:19.123495421 (385h37m18.994578456s) [upload=338446]: Done uploading segments... Dec 21 17:06:19.123517003 (385h37m18.994600048s) [upload=338446]: Upload finished. err: <nil> Duration: 6.211978901s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X.X.X.....................................................] duration: 5.851171342s trace: https://jaeger.dev.storj.io/trace/672f829e95359b44 FAIL (2025-12-21 17:46:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, Piece ID: 2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ) context deadline exceeded; piecestore: (Node ID: 125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, Piece ID: XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ) context deadline exceeded; piecestore: (Node ID: 1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7, Piece ID: MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ) context deadline exceeded; piecestore: (Node ID: 1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR, Piece ID: TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA) context deadline exceeded; piecestore: (Node ID: 1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, Piece ID: NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA) context deadline exceeded; piecestore: (Node ID: 1uzGTks5tffchQBGQK23fnWZXMfrsTHh4Zqs9UGwnWUmnQghEW, Piece ID: DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ) context deadline exceeded ======LOGS====== :06:15.347427793 (385h37m15.218510828s) [upload=338446]: Starting upload Dec 21 17:06:15.347540547 (385h37m15.218623582s) [upload=338446]: Done starting upload Dec 21 17:06:15.347708697 (385h37m15.218791732s) [upload=338446]: Uploading segments... Dec 21 17:06:15.348224540 (385h37m15.219307585s) [upload=338446]: Got next segment. Inline: false Dec 21 17:06:15.348260819 (385h37m15.219343864s) [upload=338446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 17:06:15.348266620 (385h37m15.219349665s) [upload=338446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 17:06:15.594454120 (385h37m15.465537155s) [seg_pos={0 0}, upload=338446]: Begin upload segment... Dec 21 17:06:15.594522220 (385h37m15.465605265s) [seg_pos={0 0}, upload=338446]: Done begin upload segment. Dec 21 17:06:15.594559711 (385h37m15.465642756s) [noise=true , address=145.79.227.43:4157 , piece_id=WFODGDB7OPX5HU27AOYA2G4BGNDVBJTHC2AXHVJUWT7W5S3Y53QQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.594649401 (385h37m15.465732436s) [noise=true , address=145.79.227.52:4446 , piece_id=MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595436711 (385h37m15.466519746s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595545598 (385h37m15.466628663s) [noise=true , address=145.79.227.48:4302 , piece_id=TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595937244 (385h37m15.467020289s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596027827 (385h37m15.467110872s) [noise=true , address=145.79.227.15:4876 , piece_id=NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596436265 (385h37m15.46751931s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596452004 (385h37m15.467535049s) [noise=true , address=145.79.227.49:4337 , piece_id=QATOTFHA4NAMDIGFFPZ5HEEQCC4ENPK5N436NLJ7ZDTR4F3P5CZQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596871964 (385h37m15.467955009s) [noise=true , address=145.79.227.47:4266 , piece_id=DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596893134 (385h37m15.467976169s) [noise=true , address=145.79.227.46:4602 , piece_id=CKBC5AA4MCP6Q2BZV6ELAETITPAP2WDNC5KW4YQOPWM2EDFDYNYQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597290551 (385h37m15.468373596s) [noise=true , address=145.79.227.36:4517 , piece_id=IBBYGIBCO33ESOQSSVUHNRARK36RP5VGWGFLRX2TWT55CTHQIZVA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597380512 (385h37m15.468463547s) [noise=true , address=145.79.227.2:4115 , piece_id=2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597713376 (385h37m15.468796421s) [noise=true , address=145.79.227.45:4231 , piece_id=AIKXV2DOMBCG224JP6D7F4HCEAPLTI4KVNHOUZP5HJMU775HCAPA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597875485 (385h37m15.46895852s) [noise=true , address=145.79.227.38:4977 , piece_id=XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.598258684 (385h37m15.469341729s) [noise=true , address=145.79.227.11:4732 , piece_id=ZLEFYSAEDJSJN5JAB4CLPQ6HVPSU5G6C6QM4APGO6WO5LYCF5SJQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.598373042 (385h37m15.469456077s) [noise=true , address=145.79.227.9:4366 , piece_id=3W6BSQDQBHUN32FPT4BYXWXEJ5YB7PX5PFU425TKWX6WSMEZPE6A, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:17.445367236 (385h37m17.316450281s) [upload=338446]: Next returned nil segment Dec 21 17:06:17.445397093 (385h37m17.316480128s) [upload=338446]: Waiting for error group managing segments... Dec 21 17:06:17.757291870 (385h37m17.628374905s) [noise=true , address=145.79.227.46:4602 , piece_id=CKBC5AA4MCP6Q2BZV6ELAETITPAP2WDNC5KW4YQOPWM2EDFDYNYQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.761992040 (385h37m17.633075075s) [noise=true , address=145.79.227.43:4157 , piece_id=WFODGDB7OPX5HU27AOYA2G4BGNDVBJTHC2AXHVJUWT7W5S3Y53QQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.786066502 (385h37m17.657149547s) [noise=true , address=145.79.227.11:4732 , piece_id=ZLEFYSAEDJSJN5JAB4CLPQ6HVPSU5G6C6QM4APGO6WO5LYCF5SJQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.810862998 (385h37m17.681946043s) [noise=true , address=145.79.227.49:4337 , piece_id=QATOTFHA4NAMDIGFFPZ5HEEQCC4ENPK5N436NLJ7ZDTR4F3P5CZQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.828164223 (385h37m17.699247258s) [noise=true , address=145.79.227.9:4366 , piece_id=3W6BSQDQBHUN32FPT4BYXWXEJ5YB7PX5PFU425TKWX6WSMEZPE6A, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.830694551 (385h37m17.701777586s) [noise=true , address=145.79.227.15:4876 , piece_id=NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.853562054 (385h37m17.724645089s) [noise=true , address=145.79.227.2:4115 , piece_id=2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.856593877 (385h37m17.727676912s) [noise=true , address=145.79.227.52:4446 , piece_id=MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.168473827 (385h37m18.039556872s) [noise=true , address=145.79.227.45:4231 , piece_id=AIKXV2DOMBCG224JP6D7F4HCEAPLTI4KVNHOUZP5HJMU775HCAPA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.542043663 (385h37m18.413126698s) [noise=true , address=145.79.227.47:4266 , piece_id=DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.547438796 (385h37m18.418521831s) [noise=true , address=145.79.227.38:4977 , piece_id=XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.608531947 (385h37m18.479614982s) [noise=true , address=145.79.227.36:4517 , piece_id=IBBYGIBCO33ESOQSSVUHNRARK36RP5VGWGFLRX2TWT55CTHQIZVA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.827072654 (385h37m18.698155699s) [noise=true , address=145.79.227.48:4302 , piece_id=TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.827091900 (385h37m18.698174945s) [seg_pos={0 0}, upload=338446]: Segment reached optimal threshold of 13 pieces. Dec 21 17:06:18.827253068 (385h37m18.698336103s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection Dec 21 17:06:18.827314575 (385h37m18.69839764s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827579128 (385h37m18.698662173s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: expected piece hash; context canceled; EOF Dec 21 17:06:18.827585470 (385h37m18.698668505s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827702984 (385h37m18.698786019s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 21 17:06:18.827708485 (385h37m18.69879153s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827727400 (385h37m18.698810435s) [seg_pos={0 0}, upload=338446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 17:06:18.827789619 (385h37m18.698872654s) [upload=338446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 17:06:18.827794578 (385h37m18.698877613s) [upload=338446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 17:06:19.123495421 (385h37m18.994578456s) [upload=338446]: Done uploading segments... Dec 21 17:06:19.123517003 (385h37m18.994600048s) [upload=338446]: Upload finished. err: <nil> Duration: 5.851185058s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X.X.X....................................................] duration: 5.931580434s trace: https://jaeger.dev.storj.io/trace/672f829e95359b44 FAIL (2025-12-21 17:45:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, Piece ID: 2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ) context deadline exceeded; piecestore: (Node ID: 125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, Piece ID: XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ) context deadline exceeded; piecestore: (Node ID: 1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7, Piece ID: MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ) context deadline exceeded; piecestore: (Node ID: 1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR, Piece ID: TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA) context deadline exceeded; piecestore: (Node ID: 1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, Piece ID: NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA) context deadline exceeded; piecestore: (Node ID: 1uzGTks5tffchQBGQK23fnWZXMfrsTHh4Zqs9UGwnWUmnQghEW, Piece ID: DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ) context deadline exceeded ======LOGS====== :06:15.347427793 (385h37m15.218510828s) [upload=338446]: Starting upload Dec 21 17:06:15.347540547 (385h37m15.218623582s) [upload=338446]: Done starting upload Dec 21 17:06:15.347708697 (385h37m15.218791732s) [upload=338446]: Uploading segments... Dec 21 17:06:15.348224540 (385h37m15.219307585s) [upload=338446]: Got next segment. Inline: false Dec 21 17:06:15.348260819 (385h37m15.219343864s) [upload=338446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 17:06:15.348266620 (385h37m15.219349665s) [upload=338446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 17:06:15.594454120 (385h37m15.465537155s) [seg_pos={0 0}, upload=338446]: Begin upload segment... Dec 21 17:06:15.594522220 (385h37m15.465605265s) [seg_pos={0 0}, upload=338446]: Done begin upload segment. Dec 21 17:06:15.594559711 (385h37m15.465642756s) [noise=true , address=145.79.227.43:4157 , piece_id=WFODGDB7OPX5HU27AOYA2G4BGNDVBJTHC2AXHVJUWT7W5S3Y53QQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.594649401 (385h37m15.465732436s) [noise=true , address=145.79.227.52:4446 , piece_id=MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595436711 (385h37m15.466519746s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595545598 (385h37m15.466628663s) [noise=true , address=145.79.227.48:4302 , piece_id=TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595937244 (385h37m15.467020289s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596027827 (385h37m15.467110872s) [noise=true , address=145.79.227.15:4876 , piece_id=NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596436265 (385h37m15.46751931s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596452004 (385h37m15.467535049s) [noise=true , address=145.79.227.49:4337 , piece_id=QATOTFHA4NAMDIGFFPZ5HEEQCC4ENPK5N436NLJ7ZDTR4F3P5CZQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596871964 (385h37m15.467955009s) [noise=true , address=145.79.227.47:4266 , piece_id=DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596893134 (385h37m15.467976169s) [noise=true , address=145.79.227.46:4602 , piece_id=CKBC5AA4MCP6Q2BZV6ELAETITPAP2WDNC5KW4YQOPWM2EDFDYNYQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597290551 (385h37m15.468373596s) [noise=true , address=145.79.227.36:4517 , piece_id=IBBYGIBCO33ESOQSSVUHNRARK36RP5VGWGFLRX2TWT55CTHQIZVA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597380512 (385h37m15.468463547s) [noise=true , address=145.79.227.2:4115 , piece_id=2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597713376 (385h37m15.468796421s) [noise=true , address=145.79.227.45:4231 , piece_id=AIKXV2DOMBCG224JP6D7F4HCEAPLTI4KVNHOUZP5HJMU775HCAPA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597875485 (385h37m15.46895852s) [noise=true , address=145.79.227.38:4977 , piece_id=XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.598258684 (385h37m15.469341729s) [noise=true , address=145.79.227.11:4732 , piece_id=ZLEFYSAEDJSJN5JAB4CLPQ6HVPSU5G6C6QM4APGO6WO5LYCF5SJQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.598373042 (385h37m15.469456077s) [noise=true , address=145.79.227.9:4366 , piece_id=3W6BSQDQBHUN32FPT4BYXWXEJ5YB7PX5PFU425TKWX6WSMEZPE6A, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:17.445367236 (385h37m17.316450281s) [upload=338446]: Next returned nil segment Dec 21 17:06:17.445397093 (385h37m17.316480128s) [upload=338446]: Waiting for error group managing segments... Dec 21 17:06:17.757291870 (385h37m17.628374905s) [noise=true , address=145.79.227.46:4602 , piece_id=CKBC5AA4MCP6Q2BZV6ELAETITPAP2WDNC5KW4YQOPWM2EDFDYNYQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.761992040 (385h37m17.633075075s) [noise=true , address=145.79.227.43:4157 , piece_id=WFODGDB7OPX5HU27AOYA2G4BGNDVBJTHC2AXHVJUWT7W5S3Y53QQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.786066502 (385h37m17.657149547s) [noise=true , address=145.79.227.11:4732 , piece_id=ZLEFYSAEDJSJN5JAB4CLPQ6HVPSU5G6C6QM4APGO6WO5LYCF5SJQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.810862998 (385h37m17.681946043s) [noise=true , address=145.79.227.49:4337 , piece_id=QATOTFHA4NAMDIGFFPZ5HEEQCC4ENPK5N436NLJ7ZDTR4F3P5CZQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.828164223 (385h37m17.699247258s) [noise=true , address=145.79.227.9:4366 , piece_id=3W6BSQDQBHUN32FPT4BYXWXEJ5YB7PX5PFU425TKWX6WSMEZPE6A, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.830694551 (385h37m17.701777586s) [noise=true , address=145.79.227.15:4876 , piece_id=NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.853562054 (385h37m17.724645089s) [noise=true , address=145.79.227.2:4115 , piece_id=2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.856593877 (385h37m17.727676912s) [noise=true , address=145.79.227.52:4446 , piece_id=MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.168473827 (385h37m18.039556872s) [noise=true , address=145.79.227.45:4231 , piece_id=AIKXV2DOMBCG224JP6D7F4HCEAPLTI4KVNHOUZP5HJMU775HCAPA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.542043663 (385h37m18.413126698s) [noise=true , address=145.79.227.47:4266 , piece_id=DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.547438796 (385h37m18.418521831s) [noise=true , address=145.79.227.38:4977 , piece_id=XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.608531947 (385h37m18.479614982s) [noise=true , address=145.79.227.36:4517 , piece_id=IBBYGIBCO33ESOQSSVUHNRARK36RP5VGWGFLRX2TWT55CTHQIZVA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.827072654 (385h37m18.698155699s) [noise=true , address=145.79.227.48:4302 , piece_id=TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.827091900 (385h37m18.698174945s) [seg_pos={0 0}, upload=338446]: Segment reached optimal threshold of 13 pieces. Dec 21 17:06:18.827253068 (385h37m18.698336103s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection Dec 21 17:06:18.827314575 (385h37m18.69839764s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827579128 (385h37m18.698662173s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: expected piece hash; context canceled; EOF Dec 21 17:06:18.827585470 (385h37m18.698668505s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827702984 (385h37m18.698786019s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 21 17:06:18.827708485 (385h37m18.69879153s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827727400 (385h37m18.698810435s) [seg_pos={0 0}, upload=338446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 17:06:18.827789619 (385h37m18.698872654s) [upload=338446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 17:06:18.827794578 (385h37m18.698877613s) [upload=338446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 17:06:19.123495421 (385h37m18.994578456s) [upload=338446]: Done uploading segments... Dec 21 17:06:19.123517003 (385h37m18.994600048s) [upload=338446]: Upload finished. err: <nil> Duration: 5.931606886s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X.X.X...................................................] duration: 5.847772776s trace: https://jaeger.dev.storj.io/trace/672f829e95359b44 FAIL (2025-12-21 17:45:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, Piece ID: 2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ) context deadline exceeded; piecestore: (Node ID: 125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, Piece ID: XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ) context deadline exceeded; piecestore: (Node ID: 1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7, Piece ID: MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ) context deadline exceeded; piecestore: (Node ID: 1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR, Piece ID: TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA) context deadline exceeded; piecestore: (Node ID: 1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, Piece ID: NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA) context deadline exceeded; piecestore: (Node ID: 1uzGTks5tffchQBGQK23fnWZXMfrsTHh4Zqs9UGwnWUmnQghEW, Piece ID: DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ) context deadline exceeded ======LOGS====== :06:15.347427793 (385h37m15.218510828s) [upload=338446]: Starting upload Dec 21 17:06:15.347540547 (385h37m15.218623582s) [upload=338446]: Done starting upload Dec 21 17:06:15.347708697 (385h37m15.218791732s) [upload=338446]: Uploading segments... Dec 21 17:06:15.348224540 (385h37m15.219307585s) [upload=338446]: Got next segment. Inline: false Dec 21 17:06:15.348260819 (385h37m15.219343864s) [upload=338446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 17:06:15.348266620 (385h37m15.219349665s) [upload=338446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 17:06:15.594454120 (385h37m15.465537155s) [seg_pos={0 0}, upload=338446]: Begin upload segment... Dec 21 17:06:15.594522220 (385h37m15.465605265s) [seg_pos={0 0}, upload=338446]: Done begin upload segment. Dec 21 17:06:15.594559711 (385h37m15.465642756s) [noise=true , address=145.79.227.43:4157 , piece_id=WFODGDB7OPX5HU27AOYA2G4BGNDVBJTHC2AXHVJUWT7W5S3Y53QQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.594649401 (385h37m15.465732436s) [noise=true , address=145.79.227.52:4446 , piece_id=MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595436711 (385h37m15.466519746s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595545598 (385h37m15.466628663s) [noise=true , address=145.79.227.48:4302 , piece_id=TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595937244 (385h37m15.467020289s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596027827 (385h37m15.467110872s) [noise=true , address=145.79.227.15:4876 , piece_id=NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596436265 (385h37m15.46751931s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596452004 (385h37m15.467535049s) [noise=true , address=145.79.227.49:4337 , piece_id=QATOTFHA4NAMDIGFFPZ5HEEQCC4ENPK5N436NLJ7ZDTR4F3P5CZQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596871964 (385h37m15.467955009s) [noise=true , address=145.79.227.47:4266 , piece_id=DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596893134 (385h37m15.467976169s) [noise=true , address=145.79.227.46:4602 , piece_id=CKBC5AA4MCP6Q2BZV6ELAETITPAP2WDNC5KW4YQOPWM2EDFDYNYQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597290551 (385h37m15.468373596s) [noise=true , address=145.79.227.36:4517 , piece_id=IBBYGIBCO33ESOQSSVUHNRARK36RP5VGWGFLRX2TWT55CTHQIZVA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597380512 (385h37m15.468463547s) [noise=true , address=145.79.227.2:4115 , piece_id=2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597713376 (385h37m15.468796421s) [noise=true , address=145.79.227.45:4231 , piece_id=AIKXV2DOMBCG224JP6D7F4HCEAPLTI4KVNHOUZP5HJMU775HCAPA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597875485 (385h37m15.46895852s) [noise=true , address=145.79.227.38:4977 , piece_id=XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.598258684 (385h37m15.469341729s) [noise=true , address=145.79.227.11:4732 , piece_id=ZLEFYSAEDJSJN5JAB4CLPQ6HVPSU5G6C6QM4APGO6WO5LYCF5SJQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.598373042 (385h37m15.469456077s) [noise=true , address=145.79.227.9:4366 , piece_id=3W6BSQDQBHUN32FPT4BYXWXEJ5YB7PX5PFU425TKWX6WSMEZPE6A, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:17.445367236 (385h37m17.316450281s) [upload=338446]: Next returned nil segment Dec 21 17:06:17.445397093 (385h37m17.316480128s) [upload=338446]: Waiting for error group managing segments... Dec 21 17:06:17.757291870 (385h37m17.628374905s) [noise=true , address=145.79.227.46:4602 , piece_id=CKBC5AA4MCP6Q2BZV6ELAETITPAP2WDNC5KW4YQOPWM2EDFDYNYQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.761992040 (385h37m17.633075075s) [noise=true , address=145.79.227.43:4157 , piece_id=WFODGDB7OPX5HU27AOYA2G4BGNDVBJTHC2AXHVJUWT7W5S3Y53QQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.786066502 (385h37m17.657149547s) [noise=true , address=145.79.227.11:4732 , piece_id=ZLEFYSAEDJSJN5JAB4CLPQ6HVPSU5G6C6QM4APGO6WO5LYCF5SJQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.810862998 (385h37m17.681946043s) [noise=true , address=145.79.227.49:4337 , piece_id=QATOTFHA4NAMDIGFFPZ5HEEQCC4ENPK5N436NLJ7ZDTR4F3P5CZQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.828164223 (385h37m17.699247258s) [noise=true , address=145.79.227.9:4366 , piece_id=3W6BSQDQBHUN32FPT4BYXWXEJ5YB7PX5PFU425TKWX6WSMEZPE6A, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.830694551 (385h37m17.701777586s) [noise=true , address=145.79.227.15:4876 , piece_id=NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.853562054 (385h37m17.724645089s) [noise=true , address=145.79.227.2:4115 , piece_id=2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.856593877 (385h37m17.727676912s) [noise=true , address=145.79.227.52:4446 , piece_id=MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.168473827 (385h37m18.039556872s) [noise=true , address=145.79.227.45:4231 , piece_id=AIKXV2DOMBCG224JP6D7F4HCEAPLTI4KVNHOUZP5HJMU775HCAPA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.542043663 (385h37m18.413126698s) [noise=true , address=145.79.227.47:4266 , piece_id=DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.547438796 (385h37m18.418521831s) [noise=true , address=145.79.227.38:4977 , piece_id=XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.608531947 (385h37m18.479614982s) [noise=true , address=145.79.227.36:4517 , piece_id=IBBYGIBCO33ESOQSSVUHNRARK36RP5VGWGFLRX2TWT55CTHQIZVA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.827072654 (385h37m18.698155699s) [noise=true , address=145.79.227.48:4302 , piece_id=TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.827091900 (385h37m18.698174945s) [seg_pos={0 0}, upload=338446]: Segment reached optimal threshold of 13 pieces. Dec 21 17:06:18.827253068 (385h37m18.698336103s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection Dec 21 17:06:18.827314575 (385h37m18.69839764s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827579128 (385h37m18.698662173s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: expected piece hash; context canceled; EOF Dec 21 17:06:18.827585470 (385h37m18.698668505s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827702984 (385h37m18.698786019s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 21 17:06:18.827708485 (385h37m18.69879153s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827727400 (385h37m18.698810435s) [seg_pos={0 0}, upload=338446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 17:06:18.827789619 (385h37m18.698872654s) [upload=338446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 17:06:18.827794578 (385h37m18.698877613s) [upload=338446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 17:06:19.123495421 (385h37m18.994578456s) [upload=338446]: Done uploading segments... Dec 21 17:06:19.123517003 (385h37m18.994600048s) [upload=338446]: Upload finished. err: <nil> Duration: 5.847800039s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X....X.X.X..................................................] duration: 6.062866076s trace: https://jaeger.dev.storj.io/trace/672f829e95359b44 FAIL (2025-12-21 17:44:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, Piece ID: 2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ) context deadline exceeded; piecestore: (Node ID: 125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, Piece ID: XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ) context deadline exceeded; piecestore: (Node ID: 1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7, Piece ID: MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ) context deadline exceeded; piecestore: (Node ID: 1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR, Piece ID: TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA) context deadline exceeded; piecestore: (Node ID: 1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, Piece ID: NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA) context deadline exceeded; piecestore: (Node ID: 1uzGTks5tffchQBGQK23fnWZXMfrsTHh4Zqs9UGwnWUmnQghEW, Piece ID: DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ) context deadline exceeded ======LOGS====== :06:15.347427793 (385h37m15.218510828s) [upload=338446]: Starting upload Dec 21 17:06:15.347540547 (385h37m15.218623582s) [upload=338446]: Done starting upload Dec 21 17:06:15.347708697 (385h37m15.218791732s) [upload=338446]: Uploading segments... Dec 21 17:06:15.348224540 (385h37m15.219307585s) [upload=338446]: Got next segment. Inline: false Dec 21 17:06:15.348260819 (385h37m15.219343864s) [upload=338446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 17:06:15.348266620 (385h37m15.219349665s) [upload=338446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 17:06:15.594454120 (385h37m15.465537155s) [seg_pos={0 0}, upload=338446]: Begin upload segment... Dec 21 17:06:15.594522220 (385h37m15.465605265s) [seg_pos={0 0}, upload=338446]: Done begin upload segment. Dec 21 17:06:15.594559711 (385h37m15.465642756s) [noise=true , address=145.79.227.43:4157 , piece_id=WFODGDB7OPX5HU27AOYA2G4BGNDVBJTHC2AXHVJUWT7W5S3Y53QQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.594649401 (385h37m15.465732436s) [noise=true , address=145.79.227.52:4446 , piece_id=MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595436711 (385h37m15.466519746s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595545598 (385h37m15.466628663s) [noise=true , address=145.79.227.48:4302 , piece_id=TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595937244 (385h37m15.467020289s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596027827 (385h37m15.467110872s) [noise=true , address=145.79.227.15:4876 , piece_id=NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596436265 (385h37m15.46751931s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596452004 (385h37m15.467535049s) [noise=true , address=145.79.227.49:4337 , piece_id=QATOTFHA4NAMDIGFFPZ5HEEQCC4ENPK5N436NLJ7ZDTR4F3P5CZQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596871964 (385h37m15.467955009s) [noise=true , address=145.79.227.47:4266 , piece_id=DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596893134 (385h37m15.467976169s) [noise=true , address=145.79.227.46:4602 , piece_id=CKBC5AA4MCP6Q2BZV6ELAETITPAP2WDNC5KW4YQOPWM2EDFDYNYQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597290551 (385h37m15.468373596s) [noise=true , address=145.79.227.36:4517 , piece_id=IBBYGIBCO33ESOQSSVUHNRARK36RP5VGWGFLRX2TWT55CTHQIZVA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597380512 (385h37m15.468463547s) [noise=true , address=145.79.227.2:4115 , piece_id=2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597713376 (385h37m15.468796421s) [noise=true , address=145.79.227.45:4231 , piece_id=AIKXV2DOMBCG224JP6D7F4HCEAPLTI4KVNHOUZP5HJMU775HCAPA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597875485 (385h37m15.46895852s) [noise=true , address=145.79.227.38:4977 , piece_id=XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.598258684 (385h37m15.469341729s) [noise=true , address=145.79.227.11:4732 , piece_id=ZLEFYSAEDJSJN5JAB4CLPQ6HVPSU5G6C6QM4APGO6WO5LYCF5SJQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.598373042 (385h37m15.469456077s) [noise=true , address=145.79.227.9:4366 , piece_id=3W6BSQDQBHUN32FPT4BYXWXEJ5YB7PX5PFU425TKWX6WSMEZPE6A, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:17.445367236 (385h37m17.316450281s) [upload=338446]: Next returned nil segment Dec 21 17:06:17.445397093 (385h37m17.316480128s) [upload=338446]: Waiting for error group managing segments... Dec 21 17:06:17.757291870 (385h37m17.628374905s) [noise=true , address=145.79.227.46:4602 , piece_id=CKBC5AA4MCP6Q2BZV6ELAETITPAP2WDNC5KW4YQOPWM2EDFDYNYQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.761992040 (385h37m17.633075075s) [noise=true , address=145.79.227.43:4157 , piece_id=WFODGDB7OPX5HU27AOYA2G4BGNDVBJTHC2AXHVJUWT7W5S3Y53QQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.786066502 (385h37m17.657149547s) [noise=true , address=145.79.227.11:4732 , piece_id=ZLEFYSAEDJSJN5JAB4CLPQ6HVPSU5G6C6QM4APGO6WO5LYCF5SJQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.810862998 (385h37m17.681946043s) [noise=true , address=145.79.227.49:4337 , piece_id=QATOTFHA4NAMDIGFFPZ5HEEQCC4ENPK5N436NLJ7ZDTR4F3P5CZQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.828164223 (385h37m17.699247258s) [noise=true , address=145.79.227.9:4366 , piece_id=3W6BSQDQBHUN32FPT4BYXWXEJ5YB7PX5PFU425TKWX6WSMEZPE6A, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.830694551 (385h37m17.701777586s) [noise=true , address=145.79.227.15:4876 , piece_id=NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.853562054 (385h37m17.724645089s) [noise=true , address=145.79.227.2:4115 , piece_id=2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.856593877 (385h37m17.727676912s) [noise=true , address=145.79.227.52:4446 , piece_id=MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.168473827 (385h37m18.039556872s) [noise=true , address=145.79.227.45:4231 , piece_id=AIKXV2DOMBCG224JP6D7F4HCEAPLTI4KVNHOUZP5HJMU775HCAPA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.542043663 (385h37m18.413126698s) [noise=true , address=145.79.227.47:4266 , piece_id=DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.547438796 (385h37m18.418521831s) [noise=true , address=145.79.227.38:4977 , piece_id=XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.608531947 (385h37m18.479614982s) [noise=true , address=145.79.227.36:4517 , piece_id=IBBYGIBCO33ESOQSSVUHNRARK36RP5VGWGFLRX2TWT55CTHQIZVA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.827072654 (385h37m18.698155699s) [noise=true , address=145.79.227.48:4302 , piece_id=TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.827091900 (385h37m18.698174945s) [seg_pos={0 0}, upload=338446]: Segment reached optimal threshold of 13 pieces. Dec 21 17:06:18.827253068 (385h37m18.698336103s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection Dec 21 17:06:18.827314575 (385h37m18.69839764s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827579128 (385h37m18.698662173s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: expected piece hash; context canceled; EOF Dec 21 17:06:18.827585470 (385h37m18.698668505s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827702984 (385h37m18.698786019s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 21 17:06:18.827708485 (385h37m18.69879153s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827727400 (385h37m18.698810435s) [seg_pos={0 0}, upload=338446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 17:06:18.827789619 (385h37m18.698872654s) [upload=338446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 17:06:18.827794578 (385h37m18.698877613s) [upload=338446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 17:06:19.123495421 (385h37m18.994578456s) [upload=338446]: Done uploading segments... Dec 21 17:06:19.123517003 (385h37m18.994600048s) [upload=338446]: Upload finished. err: <nil> Duration: 6.06290958s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X....X.X.X.................................................] duration: 6.022400322s trace: https://jaeger.dev.storj.io/trace/672f829e95359b44 FAIL (2025-12-21 17:44:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, Piece ID: 2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ) context deadline exceeded; piecestore: (Node ID: 125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, Piece ID: XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ) context deadline exceeded; piecestore: (Node ID: 1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7, Piece ID: MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ) context deadline exceeded; piecestore: (Node ID: 1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR, Piece ID: TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA) context deadline exceeded; piecestore: (Node ID: 1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, Piece ID: NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA) context deadline exceeded; piecestore: (Node ID: 1uzGTks5tffchQBGQK23fnWZXMfrsTHh4Zqs9UGwnWUmnQghEW, Piece ID: DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ) context deadline exceeded ======LOGS====== :06:15.347427793 (385h37m15.218510828s) [upload=338446]: Starting upload Dec 21 17:06:15.347540547 (385h37m15.218623582s) [upload=338446]: Done starting upload Dec 21 17:06:15.347708697 (385h37m15.218791732s) [upload=338446]: Uploading segments... Dec 21 17:06:15.348224540 (385h37m15.219307585s) [upload=338446]: Got next segment. Inline: false Dec 21 17:06:15.348260819 (385h37m15.219343864s) [upload=338446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 17:06:15.348266620 (385h37m15.219349665s) [upload=338446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 17:06:15.594454120 (385h37m15.465537155s) [seg_pos={0 0}, upload=338446]: Begin upload segment... Dec 21 17:06:15.594522220 (385h37m15.465605265s) [seg_pos={0 0}, upload=338446]: Done begin upload segment. Dec 21 17:06:15.594559711 (385h37m15.465642756s) [noise=true , address=145.79.227.43:4157 , piece_id=WFODGDB7OPX5HU27AOYA2G4BGNDVBJTHC2AXHVJUWT7W5S3Y53QQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.594649401 (385h37m15.465732436s) [noise=true , address=145.79.227.52:4446 , piece_id=MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595436711 (385h37m15.466519746s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595545598 (385h37m15.466628663s) [noise=true , address=145.79.227.48:4302 , piece_id=TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595937244 (385h37m15.467020289s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596027827 (385h37m15.467110872s) [noise=true , address=145.79.227.15:4876 , piece_id=NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596436265 (385h37m15.46751931s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596452004 (385h37m15.467535049s) [noise=true , address=145.79.227.49:4337 , piece_id=QATOTFHA4NAMDIGFFPZ5HEEQCC4ENPK5N436NLJ7ZDTR4F3P5CZQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596871964 (385h37m15.467955009s) [noise=true , address=145.79.227.47:4266 , piece_id=DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596893134 (385h37m15.467976169s) [noise=true , address=145.79.227.46:4602 , piece_id=CKBC5AA4MCP6Q2BZV6ELAETITPAP2WDNC5KW4YQOPWM2EDFDYNYQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597290551 (385h37m15.468373596s) [noise=true , address=145.79.227.36:4517 , piece_id=IBBYGIBCO33ESOQSSVUHNRARK36RP5VGWGFLRX2TWT55CTHQIZVA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597380512 (385h37m15.468463547s) [noise=true , address=145.79.227.2:4115 , piece_id=2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597713376 (385h37m15.468796421s) [noise=true , address=145.79.227.45:4231 , piece_id=AIKXV2DOMBCG224JP6D7F4HCEAPLTI4KVNHOUZP5HJMU775HCAPA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597875485 (385h37m15.46895852s) [noise=true , address=145.79.227.38:4977 , piece_id=XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.598258684 (385h37m15.469341729s) [noise=true , address=145.79.227.11:4732 , piece_id=ZLEFYSAEDJSJN5JAB4CLPQ6HVPSU5G6C6QM4APGO6WO5LYCF5SJQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.598373042 (385h37m15.469456077s) [noise=true , address=145.79.227.9:4366 , piece_id=3W6BSQDQBHUN32FPT4BYXWXEJ5YB7PX5PFU425TKWX6WSMEZPE6A, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:17.445367236 (385h37m17.316450281s) [upload=338446]: Next returned nil segment Dec 21 17:06:17.445397093 (385h37m17.316480128s) [upload=338446]: Waiting for error group managing segments... Dec 21 17:06:17.757291870 (385h37m17.628374905s) [noise=true , address=145.79.227.46:4602 , piece_id=CKBC5AA4MCP6Q2BZV6ELAETITPAP2WDNC5KW4YQOPWM2EDFDYNYQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.761992040 (385h37m17.633075075s) [noise=true , address=145.79.227.43:4157 , piece_id=WFODGDB7OPX5HU27AOYA2G4BGNDVBJTHC2AXHVJUWT7W5S3Y53QQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.786066502 (385h37m17.657149547s) [noise=true , address=145.79.227.11:4732 , piece_id=ZLEFYSAEDJSJN5JAB4CLPQ6HVPSU5G6C6QM4APGO6WO5LYCF5SJQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.810862998 (385h37m17.681946043s) [noise=true , address=145.79.227.49:4337 , piece_id=QATOTFHA4NAMDIGFFPZ5HEEQCC4ENPK5N436NLJ7ZDTR4F3P5CZQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.828164223 (385h37m17.699247258s) [noise=true , address=145.79.227.9:4366 , piece_id=3W6BSQDQBHUN32FPT4BYXWXEJ5YB7PX5PFU425TKWX6WSMEZPE6A, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.830694551 (385h37m17.701777586s) [noise=true , address=145.79.227.15:4876 , piece_id=NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.853562054 (385h37m17.724645089s) [noise=true , address=145.79.227.2:4115 , piece_id=2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.856593877 (385h37m17.727676912s) [noise=true , address=145.79.227.52:4446 , piece_id=MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.168473827 (385h37m18.039556872s) [noise=true , address=145.79.227.45:4231 , piece_id=AIKXV2DOMBCG224JP6D7F4HCEAPLTI4KVNHOUZP5HJMU775HCAPA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.542043663 (385h37m18.413126698s) [noise=true , address=145.79.227.47:4266 , piece_id=DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.547438796 (385h37m18.418521831s) [noise=true , address=145.79.227.38:4977 , piece_id=XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.608531947 (385h37m18.479614982s) [noise=true , address=145.79.227.36:4517 , piece_id=IBBYGIBCO33ESOQSSVUHNRARK36RP5VGWGFLRX2TWT55CTHQIZVA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.827072654 (385h37m18.698155699s) [noise=true , address=145.79.227.48:4302 , piece_id=TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.827091900 (385h37m18.698174945s) [seg_pos={0 0}, upload=338446]: Segment reached optimal threshold of 13 pieces. Dec 21 17:06:18.827253068 (385h37m18.698336103s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection Dec 21 17:06:18.827314575 (385h37m18.69839764s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827579128 (385h37m18.698662173s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: expected piece hash; context canceled; EOF Dec 21 17:06:18.827585470 (385h37m18.698668505s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827702984 (385h37m18.698786019s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 21 17:06:18.827708485 (385h37m18.69879153s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827727400 (385h37m18.698810435s) [seg_pos={0 0}, upload=338446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 17:06:18.827789619 (385h37m18.698872654s) [upload=338446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 17:06:18.827794578 (385h37m18.698877613s) [upload=338446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 17:06:19.123495421 (385h37m18.994578456s) [upload=338446]: Done uploading segments... Dec 21 17:06:19.123517003 (385h37m18.994600048s) [upload=338446]: Upload finished. err: <nil> Duration: 6.022429008s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X.X....X.X.X................................................] duration: 5.993482527s trace: https://jaeger.dev.storj.io/trace/672f829e95359b44 FAIL (2025-12-21 17:43:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, Piece ID: 2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ) context deadline exceeded; piecestore: (Node ID: 125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, Piece ID: XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ) context deadline exceeded; piecestore: (Node ID: 1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7, Piece ID: MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ) context deadline exceeded; piecestore: (Node ID: 1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR, Piece ID: TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA) context deadline exceeded; piecestore: (Node ID: 1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, Piece ID: NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA) context deadline exceeded; piecestore: (Node ID: 1uzGTks5tffchQBGQK23fnWZXMfrsTHh4Zqs9UGwnWUmnQghEW, Piece ID: DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ) context deadline exceeded ======LOGS====== :06:15.347427793 (385h37m15.218510828s) [upload=338446]: Starting upload Dec 21 17:06:15.347540547 (385h37m15.218623582s) [upload=338446]: Done starting upload Dec 21 17:06:15.347708697 (385h37m15.218791732s) [upload=338446]: Uploading segments... Dec 21 17:06:15.348224540 (385h37m15.219307585s) [upload=338446]: Got next segment. Inline: false Dec 21 17:06:15.348260819 (385h37m15.219343864s) [upload=338446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 17:06:15.348266620 (385h37m15.219349665s) [upload=338446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 17:06:15.594454120 (385h37m15.465537155s) [seg_pos={0 0}, upload=338446]: Begin upload segment... Dec 21 17:06:15.594522220 (385h37m15.465605265s) [seg_pos={0 0}, upload=338446]: Done begin upload segment. Dec 21 17:06:15.594559711 (385h37m15.465642756s) [noise=true , address=145.79.227.43:4157 , piece_id=WFODGDB7OPX5HU27AOYA2G4BGNDVBJTHC2AXHVJUWT7W5S3Y53QQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.594649401 (385h37m15.465732436s) [noise=true , address=145.79.227.52:4446 , piece_id=MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595436711 (385h37m15.466519746s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595545598 (385h37m15.466628663s) [noise=true , address=145.79.227.48:4302 , piece_id=TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595937244 (385h37m15.467020289s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596027827 (385h37m15.467110872s) [noise=true , address=145.79.227.15:4876 , piece_id=NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596436265 (385h37m15.46751931s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596452004 (385h37m15.467535049s) [noise=true , address=145.79.227.49:4337 , piece_id=QATOTFHA4NAMDIGFFPZ5HEEQCC4ENPK5N436NLJ7ZDTR4F3P5CZQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596871964 (385h37m15.467955009s) [noise=true , address=145.79.227.47:4266 , piece_id=DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596893134 (385h37m15.467976169s) [noise=true , address=145.79.227.46:4602 , piece_id=CKBC5AA4MCP6Q2BZV6ELAETITPAP2WDNC5KW4YQOPWM2EDFDYNYQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597290551 (385h37m15.468373596s) [noise=true , address=145.79.227.36:4517 , piece_id=IBBYGIBCO33ESOQSSVUHNRARK36RP5VGWGFLRX2TWT55CTHQIZVA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597380512 (385h37m15.468463547s) [noise=true , address=145.79.227.2:4115 , piece_id=2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597713376 (385h37m15.468796421s) [noise=true , address=145.79.227.45:4231 , piece_id=AIKXV2DOMBCG224JP6D7F4HCEAPLTI4KVNHOUZP5HJMU775HCAPA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597875485 (385h37m15.46895852s) [noise=true , address=145.79.227.38:4977 , piece_id=XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.598258684 (385h37m15.469341729s) [noise=true , address=145.79.227.11:4732 , piece_id=ZLEFYSAEDJSJN5JAB4CLPQ6HVPSU5G6C6QM4APGO6WO5LYCF5SJQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.598373042 (385h37m15.469456077s) [noise=true , address=145.79.227.9:4366 , piece_id=3W6BSQDQBHUN32FPT4BYXWXEJ5YB7PX5PFU425TKWX6WSMEZPE6A, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:17.445367236 (385h37m17.316450281s) [upload=338446]: Next returned nil segment Dec 21 17:06:17.445397093 (385h37m17.316480128s) [upload=338446]: Waiting for error group managing segments... Dec 21 17:06:17.757291870 (385h37m17.628374905s) [noise=true , address=145.79.227.46:4602 , piece_id=CKBC5AA4MCP6Q2BZV6ELAETITPAP2WDNC5KW4YQOPWM2EDFDYNYQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.761992040 (385h37m17.633075075s) [noise=true , address=145.79.227.43:4157 , piece_id=WFODGDB7OPX5HU27AOYA2G4BGNDVBJTHC2AXHVJUWT7W5S3Y53QQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.786066502 (385h37m17.657149547s) [noise=true , address=145.79.227.11:4732 , piece_id=ZLEFYSAEDJSJN5JAB4CLPQ6HVPSU5G6C6QM4APGO6WO5LYCF5SJQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.810862998 (385h37m17.681946043s) [noise=true , address=145.79.227.49:4337 , piece_id=QATOTFHA4NAMDIGFFPZ5HEEQCC4ENPK5N436NLJ7ZDTR4F3P5CZQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.828164223 (385h37m17.699247258s) [noise=true , address=145.79.227.9:4366 , piece_id=3W6BSQDQBHUN32FPT4BYXWXEJ5YB7PX5PFU425TKWX6WSMEZPE6A, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.830694551 (385h37m17.701777586s) [noise=true , address=145.79.227.15:4876 , piece_id=NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.853562054 (385h37m17.724645089s) [noise=true , address=145.79.227.2:4115 , piece_id=2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.856593877 (385h37m17.727676912s) [noise=true , address=145.79.227.52:4446 , piece_id=MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.168473827 (385h37m18.039556872s) [noise=true , address=145.79.227.45:4231 , piece_id=AIKXV2DOMBCG224JP6D7F4HCEAPLTI4KVNHOUZP5HJMU775HCAPA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.542043663 (385h37m18.413126698s) [noise=true , address=145.79.227.47:4266 , piece_id=DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.547438796 (385h37m18.418521831s) [noise=true , address=145.79.227.38:4977 , piece_id=XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.608531947 (385h37m18.479614982s) [noise=true , address=145.79.227.36:4517 , piece_id=IBBYGIBCO33ESOQSSVUHNRARK36RP5VGWGFLRX2TWT55CTHQIZVA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.827072654 (385h37m18.698155699s) [noise=true , address=145.79.227.48:4302 , piece_id=TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.827091900 (385h37m18.698174945s) [seg_pos={0 0}, upload=338446]: Segment reached optimal threshold of 13 pieces. Dec 21 17:06:18.827253068 (385h37m18.698336103s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection Dec 21 17:06:18.827314575 (385h37m18.69839764s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827579128 (385h37m18.698662173s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: expected piece hash; context canceled; EOF Dec 21 17:06:18.827585470 (385h37m18.698668505s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827702984 (385h37m18.698786019s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 21 17:06:18.827708485 (385h37m18.69879153s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827727400 (385h37m18.698810435s) [seg_pos={0 0}, upload=338446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 17:06:18.827789619 (385h37m18.698872654s) [upload=338446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 17:06:18.827794578 (385h37m18.698877613s) [upload=338446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 17:06:19.123495421 (385h37m18.994578456s) [upload=338446]: Done uploading segments... Dec 21 17:06:19.123517003 (385h37m18.994600048s) [upload=338446]: Upload finished. err: <nil> Duration: 5.993514016s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X.X....X.X.X...............................................] duration: 5.75802715s trace: https://jaeger.dev.storj.io/trace/672f829e95359b44 FAIL (2025-12-21 17:42:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, Piece ID: 2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ) context deadline exceeded; piecestore: (Node ID: 125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, Piece ID: XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ) context deadline exceeded; piecestore: (Node ID: 1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7, Piece ID: MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ) context deadline exceeded; piecestore: (Node ID: 1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR, Piece ID: TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA) context deadline exceeded; piecestore: (Node ID: 1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, Piece ID: NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA) context deadline exceeded; piecestore: (Node ID: 1uzGTks5tffchQBGQK23fnWZXMfrsTHh4Zqs9UGwnWUmnQghEW, Piece ID: DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ) context deadline exceeded ======LOGS====== :06:15.347427793 (385h37m15.218510828s) [upload=338446]: Starting upload Dec 21 17:06:15.347540547 (385h37m15.218623582s) [upload=338446]: Done starting upload Dec 21 17:06:15.347708697 (385h37m15.218791732s) [upload=338446]: Uploading segments... Dec 21 17:06:15.348224540 (385h37m15.219307585s) [upload=338446]: Got next segment. Inline: false Dec 21 17:06:15.348260819 (385h37m15.219343864s) [upload=338446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 17:06:15.348266620 (385h37m15.219349665s) [upload=338446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 17:06:15.594454120 (385h37m15.465537155s) [seg_pos={0 0}, upload=338446]: Begin upload segment... Dec 21 17:06:15.594522220 (385h37m15.465605265s) [seg_pos={0 0}, upload=338446]: Done begin upload segment. Dec 21 17:06:15.594559711 (385h37m15.465642756s) [noise=true , address=145.79.227.43:4157 , piece_id=WFODGDB7OPX5HU27AOYA2G4BGNDVBJTHC2AXHVJUWT7W5S3Y53QQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.594649401 (385h37m15.465732436s) [noise=true , address=145.79.227.52:4446 , piece_id=MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595436711 (385h37m15.466519746s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595545598 (385h37m15.466628663s) [noise=true , address=145.79.227.48:4302 , piece_id=TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595937244 (385h37m15.467020289s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596027827 (385h37m15.467110872s) [noise=true , address=145.79.227.15:4876 , piece_id=NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596436265 (385h37m15.46751931s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596452004 (385h37m15.467535049s) [noise=true , address=145.79.227.49:4337 , piece_id=QATOTFHA4NAMDIGFFPZ5HEEQCC4ENPK5N436NLJ7ZDTR4F3P5CZQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596871964 (385h37m15.467955009s) [noise=true , address=145.79.227.47:4266 , piece_id=DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596893134 (385h37m15.467976169s) [noise=true , address=145.79.227.46:4602 , piece_id=CKBC5AA4MCP6Q2BZV6ELAETITPAP2WDNC5KW4YQOPWM2EDFDYNYQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597290551 (385h37m15.468373596s) [noise=true , address=145.79.227.36:4517 , piece_id=IBBYGIBCO33ESOQSSVUHNRARK36RP5VGWGFLRX2TWT55CTHQIZVA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597380512 (385h37m15.468463547s) [noise=true , address=145.79.227.2:4115 , piece_id=2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597713376 (385h37m15.468796421s) [noise=true , address=145.79.227.45:4231 , piece_id=AIKXV2DOMBCG224JP6D7F4HCEAPLTI4KVNHOUZP5HJMU775HCAPA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597875485 (385h37m15.46895852s) [noise=true , address=145.79.227.38:4977 , piece_id=XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.598258684 (385h37m15.469341729s) [noise=true , address=145.79.227.11:4732 , piece_id=ZLEFYSAEDJSJN5JAB4CLPQ6HVPSU5G6C6QM4APGO6WO5LYCF5SJQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.598373042 (385h37m15.469456077s) [noise=true , address=145.79.227.9:4366 , piece_id=3W6BSQDQBHUN32FPT4BYXWXEJ5YB7PX5PFU425TKWX6WSMEZPE6A, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:17.445367236 (385h37m17.316450281s) [upload=338446]: Next returned nil segment Dec 21 17:06:17.445397093 (385h37m17.316480128s) [upload=338446]: Waiting for error group managing segments... Dec 21 17:06:17.757291870 (385h37m17.628374905s) [noise=true , address=145.79.227.46:4602 , piece_id=CKBC5AA4MCP6Q2BZV6ELAETITPAP2WDNC5KW4YQOPWM2EDFDYNYQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.761992040 (385h37m17.633075075s) [noise=true , address=145.79.227.43:4157 , piece_id=WFODGDB7OPX5HU27AOYA2G4BGNDVBJTHC2AXHVJUWT7W5S3Y53QQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.786066502 (385h37m17.657149547s) [noise=true , address=145.79.227.11:4732 , piece_id=ZLEFYSAEDJSJN5JAB4CLPQ6HVPSU5G6C6QM4APGO6WO5LYCF5SJQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.810862998 (385h37m17.681946043s) [noise=true , address=145.79.227.49:4337 , piece_id=QATOTFHA4NAMDIGFFPZ5HEEQCC4ENPK5N436NLJ7ZDTR4F3P5CZQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.828164223 (385h37m17.699247258s) [noise=true , address=145.79.227.9:4366 , piece_id=3W6BSQDQBHUN32FPT4BYXWXEJ5YB7PX5PFU425TKWX6WSMEZPE6A, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.830694551 (385h37m17.701777586s) [noise=true , address=145.79.227.15:4876 , piece_id=NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.853562054 (385h37m17.724645089s) [noise=true , address=145.79.227.2:4115 , piece_id=2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.856593877 (385h37m17.727676912s) [noise=true , address=145.79.227.52:4446 , piece_id=MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.168473827 (385h37m18.039556872s) [noise=true , address=145.79.227.45:4231 , piece_id=AIKXV2DOMBCG224JP6D7F4HCEAPLTI4KVNHOUZP5HJMU775HCAPA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.542043663 (385h37m18.413126698s) [noise=true , address=145.79.227.47:4266 , piece_id=DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.547438796 (385h37m18.418521831s) [noise=true , address=145.79.227.38:4977 , piece_id=XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.608531947 (385h37m18.479614982s) [noise=true , address=145.79.227.36:4517 , piece_id=IBBYGIBCO33ESOQSSVUHNRARK36RP5VGWGFLRX2TWT55CTHQIZVA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.827072654 (385h37m18.698155699s) [noise=true , address=145.79.227.48:4302 , piece_id=TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.827091900 (385h37m18.698174945s) [seg_pos={0 0}, upload=338446]: Segment reached optimal threshold of 13 pieces. Dec 21 17:06:18.827253068 (385h37m18.698336103s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection Dec 21 17:06:18.827314575 (385h37m18.69839764s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827579128 (385h37m18.698662173s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: expected piece hash; context canceled; EOF Dec 21 17:06:18.827585470 (385h37m18.698668505s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827702984 (385h37m18.698786019s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 21 17:06:18.827708485 (385h37m18.69879153s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827727400 (385h37m18.698810435s) [seg_pos={0 0}, upload=338446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 17:06:18.827789619 (385h37m18.698872654s) [upload=338446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 17:06:18.827794578 (385h37m18.698877613s) [upload=338446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 17:06:19.123495421 (385h37m18.994578456s) [upload=338446]: Done uploading segments... Dec 21 17:06:19.123517003 (385h37m18.994600048s) [upload=338446]: Upload finished. err: <nil> Duration: 5.758051918s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X.X.X....X.X.X..............................................] duration: 5.840767894s trace: https://jaeger.dev.storj.io/trace/672f829e95359b44 FAIL (2025-12-21 17:41:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, Piece ID: 2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ) context deadline exceeded; piecestore: (Node ID: 125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, Piece ID: XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ) context deadline exceeded; piecestore: (Node ID: 1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7, Piece ID: MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ) context deadline exceeded; piecestore: (Node ID: 1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR, Piece ID: TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA) context deadline exceeded; piecestore: (Node ID: 1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, Piece ID: NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA) context deadline exceeded; piecestore: (Node ID: 1uzGTks5tffchQBGQK23fnWZXMfrsTHh4Zqs9UGwnWUmnQghEW, Piece ID: DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ) context deadline exceeded ======LOGS====== :06:15.347427793 (385h37m15.218510828s) [upload=338446]: Starting upload Dec 21 17:06:15.347540547 (385h37m15.218623582s) [upload=338446]: Done starting upload Dec 21 17:06:15.347708697 (385h37m15.218791732s) [upload=338446]: Uploading segments... Dec 21 17:06:15.348224540 (385h37m15.219307585s) [upload=338446]: Got next segment. Inline: false Dec 21 17:06:15.348260819 (385h37m15.219343864s) [upload=338446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 17:06:15.348266620 (385h37m15.219349665s) [upload=338446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 17:06:15.594454120 (385h37m15.465537155s) [seg_pos={0 0}, upload=338446]: Begin upload segment... Dec 21 17:06:15.594522220 (385h37m15.465605265s) [seg_pos={0 0}, upload=338446]: Done begin upload segment. Dec 21 17:06:15.594559711 (385h37m15.465642756s) [noise=true , address=145.79.227.43:4157 , piece_id=WFODGDB7OPX5HU27AOYA2G4BGNDVBJTHC2AXHVJUWT7W5S3Y53QQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.594649401 (385h37m15.465732436s) [noise=true , address=145.79.227.52:4446 , piece_id=MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595436711 (385h37m15.466519746s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595545598 (385h37m15.466628663s) [noise=true , address=145.79.227.48:4302 , piece_id=TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595937244 (385h37m15.467020289s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596027827 (385h37m15.467110872s) [noise=true , address=145.79.227.15:4876 , piece_id=NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596436265 (385h37m15.46751931s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596452004 (385h37m15.467535049s) [noise=true , address=145.79.227.49:4337 , piece_id=QATOTFHA4NAMDIGFFPZ5HEEQCC4ENPK5N436NLJ7ZDTR4F3P5CZQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596871964 (385h37m15.467955009s) [noise=true , address=145.79.227.47:4266 , piece_id=DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596893134 (385h37m15.467976169s) [noise=true , address=145.79.227.46:4602 , piece_id=CKBC5AA4MCP6Q2BZV6ELAETITPAP2WDNC5KW4YQOPWM2EDFDYNYQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597290551 (385h37m15.468373596s) [noise=true , address=145.79.227.36:4517 , piece_id=IBBYGIBCO33ESOQSSVUHNRARK36RP5VGWGFLRX2TWT55CTHQIZVA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597380512 (385h37m15.468463547s) [noise=true , address=145.79.227.2:4115 , piece_id=2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597713376 (385h37m15.468796421s) [noise=true , address=145.79.227.45:4231 , piece_id=AIKXV2DOMBCG224JP6D7F4HCEAPLTI4KVNHOUZP5HJMU775HCAPA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597875485 (385h37m15.46895852s) [noise=true , address=145.79.227.38:4977 , piece_id=XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.598258684 (385h37m15.469341729s) [noise=true , address=145.79.227.11:4732 , piece_id=ZLEFYSAEDJSJN5JAB4CLPQ6HVPSU5G6C6QM4APGO6WO5LYCF5SJQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.598373042 (385h37m15.469456077s) [noise=true , address=145.79.227.9:4366 , piece_id=3W6BSQDQBHUN32FPT4BYXWXEJ5YB7PX5PFU425TKWX6WSMEZPE6A, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:17.445367236 (385h37m17.316450281s) [upload=338446]: Next returned nil segment Dec 21 17:06:17.445397093 (385h37m17.316480128s) [upload=338446]: Waiting for error group managing segments... Dec 21 17:06:17.757291870 (385h37m17.628374905s) [noise=true , address=145.79.227.46:4602 , piece_id=CKBC5AA4MCP6Q2BZV6ELAETITPAP2WDNC5KW4YQOPWM2EDFDYNYQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.761992040 (385h37m17.633075075s) [noise=true , address=145.79.227.43:4157 , piece_id=WFODGDB7OPX5HU27AOYA2G4BGNDVBJTHC2AXHVJUWT7W5S3Y53QQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.786066502 (385h37m17.657149547s) [noise=true , address=145.79.227.11:4732 , piece_id=ZLEFYSAEDJSJN5JAB4CLPQ6HVPSU5G6C6QM4APGO6WO5LYCF5SJQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.810862998 (385h37m17.681946043s) [noise=true , address=145.79.227.49:4337 , piece_id=QATOTFHA4NAMDIGFFPZ5HEEQCC4ENPK5N436NLJ7ZDTR4F3P5CZQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.828164223 (385h37m17.699247258s) [noise=true , address=145.79.227.9:4366 , piece_id=3W6BSQDQBHUN32FPT4BYXWXEJ5YB7PX5PFU425TKWX6WSMEZPE6A, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.830694551 (385h37m17.701777586s) [noise=true , address=145.79.227.15:4876 , piece_id=NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.853562054 (385h37m17.724645089s) [noise=true , address=145.79.227.2:4115 , piece_id=2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.856593877 (385h37m17.727676912s) [noise=true , address=145.79.227.52:4446 , piece_id=MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.168473827 (385h37m18.039556872s) [noise=true , address=145.79.227.45:4231 , piece_id=AIKXV2DOMBCG224JP6D7F4HCEAPLTI4KVNHOUZP5HJMU775HCAPA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.542043663 (385h37m18.413126698s) [noise=true , address=145.79.227.47:4266 , piece_id=DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.547438796 (385h37m18.418521831s) [noise=true , address=145.79.227.38:4977 , piece_id=XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.608531947 (385h37m18.479614982s) [noise=true , address=145.79.227.36:4517 , piece_id=IBBYGIBCO33ESOQSSVUHNRARK36RP5VGWGFLRX2TWT55CTHQIZVA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.827072654 (385h37m18.698155699s) [noise=true , address=145.79.227.48:4302 , piece_id=TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.827091900 (385h37m18.698174945s) [seg_pos={0 0}, upload=338446]: Segment reached optimal threshold of 13 pieces. Dec 21 17:06:18.827253068 (385h37m18.698336103s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection Dec 21 17:06:18.827314575 (385h37m18.69839764s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827579128 (385h37m18.698662173s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: expected piece hash; context canceled; EOF Dec 21 17:06:18.827585470 (385h37m18.698668505s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827702984 (385h37m18.698786019s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 21 17:06:18.827708485 (385h37m18.69879153s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827727400 (385h37m18.698810435s) [seg_pos={0 0}, upload=338446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 17:06:18.827789619 (385h37m18.698872654s) [upload=338446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 17:06:18.827794578 (385h37m18.698877613s) [upload=338446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 17:06:19.123495421 (385h37m18.994578456s) [upload=338446]: Done uploading segments... Dec 21 17:06:19.123517003 (385h37m18.994600048s) [upload=338446]: Upload finished. err: <nil> Duration: 5.840801056s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X.X.X....X.X.X.............................................] duration: 21.579448016s trace: https://jaeger.dev.storj.io/trace/672f829e95359b44 FAIL (2025-12-21 17:40:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, Piece ID: 2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ) context deadline exceeded; piecestore: (Node ID: 125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, Piece ID: XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ) context deadline exceeded; piecestore: (Node ID: 1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7, Piece ID: MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ) context deadline exceeded; piecestore: (Node ID: 1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR, Piece ID: TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA) context deadline exceeded; piecestore: (Node ID: 1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, Piece ID: NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA) context deadline exceeded; piecestore: (Node ID: 1uzGTks5tffchQBGQK23fnWZXMfrsTHh4Zqs9UGwnWUmnQghEW, Piece ID: DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ) context deadline exceeded ======LOGS====== :06:15.347427793 (385h37m15.218510828s) [upload=338446]: Starting upload Dec 21 17:06:15.347540547 (385h37m15.218623582s) [upload=338446]: Done starting upload Dec 21 17:06:15.347708697 (385h37m15.218791732s) [upload=338446]: Uploading segments... Dec 21 17:06:15.348224540 (385h37m15.219307585s) [upload=338446]: Got next segment. Inline: false Dec 21 17:06:15.348260819 (385h37m15.219343864s) [upload=338446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 17:06:15.348266620 (385h37m15.219349665s) [upload=338446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 17:06:15.594454120 (385h37m15.465537155s) [seg_pos={0 0}, upload=338446]: Begin upload segment... Dec 21 17:06:15.594522220 (385h37m15.465605265s) [seg_pos={0 0}, upload=338446]: Done begin upload segment. Dec 21 17:06:15.594559711 (385h37m15.465642756s) [noise=true , address=145.79.227.43:4157 , piece_id=WFODGDB7OPX5HU27AOYA2G4BGNDVBJTHC2AXHVJUWT7W5S3Y53QQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.594649401 (385h37m15.465732436s) [noise=true , address=145.79.227.52:4446 , piece_id=MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595436711 (385h37m15.466519746s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595545598 (385h37m15.466628663s) [noise=true , address=145.79.227.48:4302 , piece_id=TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595937244 (385h37m15.467020289s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596027827 (385h37m15.467110872s) [noise=true , address=145.79.227.15:4876 , piece_id=NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596436265 (385h37m15.46751931s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596452004 (385h37m15.467535049s) [noise=true , address=145.79.227.49:4337 , piece_id=QATOTFHA4NAMDIGFFPZ5HEEQCC4ENPK5N436NLJ7ZDTR4F3P5CZQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596871964 (385h37m15.467955009s) [noise=true , address=145.79.227.47:4266 , piece_id=DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596893134 (385h37m15.467976169s) [noise=true , address=145.79.227.46:4602 , piece_id=CKBC5AA4MCP6Q2BZV6ELAETITPAP2WDNC5KW4YQOPWM2EDFDYNYQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597290551 (385h37m15.468373596s) [noise=true , address=145.79.227.36:4517 , piece_id=IBBYGIBCO33ESOQSSVUHNRARK36RP5VGWGFLRX2TWT55CTHQIZVA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597380512 (385h37m15.468463547s) [noise=true , address=145.79.227.2:4115 , piece_id=2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597713376 (385h37m15.468796421s) [noise=true , address=145.79.227.45:4231 , piece_id=AIKXV2DOMBCG224JP6D7F4HCEAPLTI4KVNHOUZP5HJMU775HCAPA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597875485 (385h37m15.46895852s) [noise=true , address=145.79.227.38:4977 , piece_id=XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.598258684 (385h37m15.469341729s) [noise=true , address=145.79.227.11:4732 , piece_id=ZLEFYSAEDJSJN5JAB4CLPQ6HVPSU5G6C6QM4APGO6WO5LYCF5SJQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.598373042 (385h37m15.469456077s) [noise=true , address=145.79.227.9:4366 , piece_id=3W6BSQDQBHUN32FPT4BYXWXEJ5YB7PX5PFU425TKWX6WSMEZPE6A, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:17.445367236 (385h37m17.316450281s) [upload=338446]: Next returned nil segment Dec 21 17:06:17.445397093 (385h37m17.316480128s) [upload=338446]: Waiting for error group managing segments... Dec 21 17:06:17.757291870 (385h37m17.628374905s) [noise=true , address=145.79.227.46:4602 , piece_id=CKBC5AA4MCP6Q2BZV6ELAETITPAP2WDNC5KW4YQOPWM2EDFDYNYQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.761992040 (385h37m17.633075075s) [noise=true , address=145.79.227.43:4157 , piece_id=WFODGDB7OPX5HU27AOYA2G4BGNDVBJTHC2AXHVJUWT7W5S3Y53QQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.786066502 (385h37m17.657149547s) [noise=true , address=145.79.227.11:4732 , piece_id=ZLEFYSAEDJSJN5JAB4CLPQ6HVPSU5G6C6QM4APGO6WO5LYCF5SJQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.810862998 (385h37m17.681946043s) [noise=true , address=145.79.227.49:4337 , piece_id=QATOTFHA4NAMDIGFFPZ5HEEQCC4ENPK5N436NLJ7ZDTR4F3P5CZQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.828164223 (385h37m17.699247258s) [noise=true , address=145.79.227.9:4366 , piece_id=3W6BSQDQBHUN32FPT4BYXWXEJ5YB7PX5PFU425TKWX6WSMEZPE6A, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.830694551 (385h37m17.701777586s) [noise=true , address=145.79.227.15:4876 , piece_id=NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.853562054 (385h37m17.724645089s) [noise=true , address=145.79.227.2:4115 , piece_id=2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.856593877 (385h37m17.727676912s) [noise=true , address=145.79.227.52:4446 , piece_id=MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.168473827 (385h37m18.039556872s) [noise=true , address=145.79.227.45:4231 , piece_id=AIKXV2DOMBCG224JP6D7F4HCEAPLTI4KVNHOUZP5HJMU775HCAPA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.542043663 (385h37m18.413126698s) [noise=true , address=145.79.227.47:4266 , piece_id=DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.547438796 (385h37m18.418521831s) [noise=true , address=145.79.227.38:4977 , piece_id=XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.608531947 (385h37m18.479614982s) [noise=true , address=145.79.227.36:4517 , piece_id=IBBYGIBCO33ESOQSSVUHNRARK36RP5VGWGFLRX2TWT55CTHQIZVA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.827072654 (385h37m18.698155699s) [noise=true , address=145.79.227.48:4302 , piece_id=TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.827091900 (385h37m18.698174945s) [seg_pos={0 0}, upload=338446]: Segment reached optimal threshold of 13 pieces. Dec 21 17:06:18.827253068 (385h37m18.698336103s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection Dec 21 17:06:18.827314575 (385h37m18.69839764s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827579128 (385h37m18.698662173s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: expected piece hash; context canceled; EOF Dec 21 17:06:18.827585470 (385h37m18.698668505s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827702984 (385h37m18.698786019s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 21 17:06:18.827708485 (385h37m18.69879153s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827727400 (385h37m18.698810435s) [seg_pos={0 0}, upload=338446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 17:06:18.827789619 (385h37m18.698872654s) [upload=338446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 17:06:18.827794578 (385h37m18.698877613s) [upload=338446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 17:06:19.123495421 (385h37m18.994578456s) [upload=338446]: Done uploading segments... Dec 21 17:06:19.123517003 (385h37m18.994600048s) [upload=338446]: Upload finished. err: <nil> Duration: 21.57947687s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X.X.X....X.X.X............................................] duration: 6.244434229s trace: https://jaeger.dev.storj.io/trace/672f829e95359b44 FAIL (2025-12-21 17:39:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, Piece ID: 2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ) context deadline exceeded; piecestore: (Node ID: 125dJwj1tvBkFrgDCoj3FhpH553yGgjxPuSTGxbrkGeDwdKk6M8, Piece ID: XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ) context deadline exceeded; piecestore: (Node ID: 1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7, Piece ID: MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ) context deadline exceeded; piecestore: (Node ID: 1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR, Piece ID: TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA) context deadline exceeded; piecestore: (Node ID: 1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, Piece ID: NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA) context deadline exceeded; piecestore: (Node ID: 1uzGTks5tffchQBGQK23fnWZXMfrsTHh4Zqs9UGwnWUmnQghEW, Piece ID: DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ) context deadline exceeded ======LOGS====== :06:15.347427793 (385h37m15.218510828s) [upload=338446]: Starting upload Dec 21 17:06:15.347540547 (385h37m15.218623582s) [upload=338446]: Done starting upload Dec 21 17:06:15.347708697 (385h37m15.218791732s) [upload=338446]: Uploading segments... Dec 21 17:06:15.348224540 (385h37m15.219307585s) [upload=338446]: Got next segment. Inline: false Dec 21 17:06:15.348260819 (385h37m15.219343864s) [upload=338446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 17:06:15.348266620 (385h37m15.219349665s) [upload=338446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 17:06:15.594454120 (385h37m15.465537155s) [seg_pos={0 0}, upload=338446]: Begin upload segment... Dec 21 17:06:15.594522220 (385h37m15.465605265s) [seg_pos={0 0}, upload=338446]: Done begin upload segment. Dec 21 17:06:15.594559711 (385h37m15.465642756s) [noise=true , address=145.79.227.43:4157 , piece_id=WFODGDB7OPX5HU27AOYA2G4BGNDVBJTHC2AXHVJUWT7W5S3Y53QQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.594649401 (385h37m15.465732436s) [noise=true , address=145.79.227.52:4446 , piece_id=MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595436711 (385h37m15.466519746s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595545598 (385h37m15.466628663s) [noise=true , address=145.79.227.48:4302 , piece_id=TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.595937244 (385h37m15.467020289s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596027827 (385h37m15.467110872s) [noise=true , address=145.79.227.15:4876 , piece_id=NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596436265 (385h37m15.46751931s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596452004 (385h37m15.467535049s) [noise=true , address=145.79.227.49:4337 , piece_id=QATOTFHA4NAMDIGFFPZ5HEEQCC4ENPK5N436NLJ7ZDTR4F3P5CZQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596871964 (385h37m15.467955009s) [noise=true , address=145.79.227.47:4266 , piece_id=DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.596893134 (385h37m15.467976169s) [noise=true , address=145.79.227.46:4602 , piece_id=CKBC5AA4MCP6Q2BZV6ELAETITPAP2WDNC5KW4YQOPWM2EDFDYNYQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597290551 (385h37m15.468373596s) [noise=true , address=145.79.227.36:4517 , piece_id=IBBYGIBCO33ESOQSSVUHNRARK36RP5VGWGFLRX2TWT55CTHQIZVA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597380512 (385h37m15.468463547s) [noise=true , address=145.79.227.2:4115 , piece_id=2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597713376 (385h37m15.468796421s) [noise=true , address=145.79.227.45:4231 , piece_id=AIKXV2DOMBCG224JP6D7F4HCEAPLTI4KVNHOUZP5HJMU775HCAPA, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.597875485 (385h37m15.46895852s) [noise=true , address=145.79.227.38:4977 , piece_id=XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.598258684 (385h37m15.469341729s) [noise=true , address=145.79.227.11:4732 , piece_id=ZLEFYSAEDJSJN5JAB4CLPQ6HVPSU5G6C6QM4APGO6WO5LYCF5SJQ, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:15.598373042 (385h37m15.469456077s) [noise=true , address=145.79.227.9:4366 , piece_id=3W6BSQDQBHUN32FPT4BYXWXEJ5YB7PX5PFU425TKWX6WSMEZPE6A, seg_pos={0 0}, upload=338446]: Uploading piece... Dec 21 17:06:17.445367236 (385h37m17.316450281s) [upload=338446]: Next returned nil segment Dec 21 17:06:17.445397093 (385h37m17.316480128s) [upload=338446]: Waiting for error group managing segments... Dec 21 17:06:17.757291870 (385h37m17.628374905s) [noise=true , address=145.79.227.46:4602 , piece_id=CKBC5AA4MCP6Q2BZV6ELAETITPAP2WDNC5KW4YQOPWM2EDFDYNYQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.761992040 (385h37m17.633075075s) [noise=true , address=145.79.227.43:4157 , piece_id=WFODGDB7OPX5HU27AOYA2G4BGNDVBJTHC2AXHVJUWT7W5S3Y53QQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.786066502 (385h37m17.657149547s) [noise=true , address=145.79.227.11:4732 , piece_id=ZLEFYSAEDJSJN5JAB4CLPQ6HVPSU5G6C6QM4APGO6WO5LYCF5SJQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.810862998 (385h37m17.681946043s) [noise=true , address=145.79.227.49:4337 , piece_id=QATOTFHA4NAMDIGFFPZ5HEEQCC4ENPK5N436NLJ7ZDTR4F3P5CZQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.828164223 (385h37m17.699247258s) [noise=true , address=145.79.227.9:4366 , piece_id=3W6BSQDQBHUN32FPT4BYXWXEJ5YB7PX5PFU425TKWX6WSMEZPE6A, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.830694551 (385h37m17.701777586s) [noise=true , address=145.79.227.15:4876 , piece_id=NH2V44NVYKOXDHZ74HH75DIGG7FEPZ5O2IZHXVRXUKLJSSQBLTCA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.853562054 (385h37m17.724645089s) [noise=true , address=145.79.227.2:4115 , piece_id=2R35VLV25GFGUF2I3UCALJ7SO2MWW3AJA4DKFOHTH6WUN22DOMOQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:17.856593877 (385h37m17.727676912s) [noise=true , address=145.79.227.52:4446 , piece_id=MU5FDAOZ2HVIYVUP2RDPYGRZLZR7CVFFEIGJYR7LF254BUPAHNUQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.168473827 (385h37m18.039556872s) [noise=true , address=145.79.227.45:4231 , piece_id=AIKXV2DOMBCG224JP6D7F4HCEAPLTI4KVNHOUZP5HJMU775HCAPA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.542043663 (385h37m18.413126698s) [noise=true , address=145.79.227.47:4266 , piece_id=DQQA22LQBHK5SSSWXDJX2RLH5IGGRI6T7KW2VGFB5URN27JA6IIQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.547438796 (385h37m18.418521831s) [noise=true , address=145.79.227.38:4977 , piece_id=XJ23M6RD7YHFU2OR26CCVKHPE5Q4EQXTYZA3FEH3JQ7BAN6S7QFQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.608531947 (385h37m18.479614982s) [noise=true , address=145.79.227.36:4517 , piece_id=IBBYGIBCO33ESOQSSVUHNRARK36RP5VGWGFLRX2TWT55CTHQIZVA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.827072654 (385h37m18.698155699s) [noise=true , address=145.79.227.48:4302 , piece_id=TD2NDJBX7EAUHJWZYX2EZD25RQ73MZ474AFHABDI4ARCONP55DPA, seg_pos={0 0}, upload=338446]: Done uploading piece. err: <nil> Dec 21 17:06:18.827091900 (385h37m18.698174945s) [seg_pos={0 0}, upload=338446]: Segment reached optimal threshold of 13 pieces. Dec 21 17:06:18.827253068 (385h37m18.698336103s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE): protocol: write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection; write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37006->145.79.227.39:4015: use of closed network connection Dec 21 17:06:18.827314575 (385h37m18.69839764s) [noise=true , address=145.79.227.39:4015 , piece_id=NHCWSYUB53T6OGJ76KEZ2TJQNG72GHUJW2JFYPDYA25ZLC5GDD3Q, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827579128 (385h37m18.698662173s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1G3fe4SHMdNAjAmqLNyFb6qDKL94aAWGYE9qhczKmozKFhKKEV): protocol: expected piece hash; context canceled; EOF Dec 21 17:06:18.827585470 (385h37m18.698668505s) [noise=true , address=145.79.227.20:4437 , piece_id=APOOEEICJP2T7ZPF3Z3KMHPBH3MJLKE46XENFZXP5LGWQ756VLVQ, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827702984 (385h37m18.698786019s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 21 17:06:18.827708485 (385h37m18.69879153s) [noise=true , address=145.79.227.40:4051 , piece_id=MWCOTABCMY567CXOKIUHDRNUEKKAPC5YLV3ODPUZZ2M72SLHVE4A, seg_pos={0 0}, upload=338446]: Long tail cancellation due to optimal threshold Dec 21 17:06:18.827727400 (385h37m18.698810435s) [seg_pos={0 0}, upload=338446]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 17:06:18.827789619 (385h37m18.698872654s) [upload=338446]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 17:06:18.827794578 (385h37m18.698877613s) [upload=338446]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 17:06:19.123495421 (385h37m18.994578456s) [upload=338446]: Done uploading segments... Dec 21 17:06:19.123517003 (385h37m18.994600048s) [upload=338446]: Upload finished. err: <nil> Duration: 6.244458227s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)