Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [XXX.X.X.XX.X.XXX...X.X.XX.XXX....X...X.X.X...XXXXX.X.XXX.X..] duration: 5.967756741s trace: https://jaeger.dev.storj.io/trace/43b8ae0fc43abb1d FAIL (2025-12-28 00:56:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 00:54:39.571036892 (537h25m39.442119937s) [upload=471336]: Starting upload Dec 28 00:54:39.571112146 (537h25m39.442195181s) [upload=471336]: Done starting upload Dec 28 00:54:39.571146471 (537h25m39.442229516s) [upload=471336]: Uploading segments... Dec 28 00:54:39.571352564 (537h25m39.442435609s) [upload=471336]: Got next segment. Inline: false Dec 28 00:54:39.571389333 (537h25m39.442472379s) [upload=471336]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:54:39.571394974 (537h25m39.442478019s) [upload=471336]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:54:39.819105876 (537h25m39.690188922s) [seg_pos={0 0}, upload=471336]: Begin upload segment... Dec 28 00:54:39.819160561 (537h25m39.690243596s) [seg_pos={0 0}, upload=471336]: Done begin upload segment. Dec 28 00:54:39.819212550 (537h25m39.690295585s) [noise=true , address=145.79.227.42:4122 , piece_id=4WUUPI5HFJVNLZWQVGJ6PEP3HOZ42Z5FXRKUTZWUJT3UIZFEJ5LA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.819219323 (537h25m39.690302358s) [noise=true , address=77.90.31.12:4590 , piece_id=Y4OAPBHUU4OMR5BK5UMI7KMXSPKFMNC36ESCVIAHRU7LUSM6BCFA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.819792885 (537h25m39.69087592s) [noise=true , address=45.91.116.7:4011 , piece_id=OQWCFH43GXQSXQOUWZ66OFOH5FLEPKIMBZOPYN5FOEBZMMDLNUBA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.820079420 (537h25m39.691162455s) [noise=true , address=145.79.227.48:4301 , piece_id=GVYJFKDHEIYL6JCGOENR3QMPVHZEOQTGD7SHKJNLIDIPFD4YIPQA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.820343273 (537h25m39.691426308s) [noise=true , address=145.79.227.19:4077 , piece_id=LOVU5N3YZLT5R2EZ37STPIIWWQD4B6PT3XHNOEK6LPGMSVZPRNFQ, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.820600693 (537h25m39.691683728s) [noise=true , address=145.79.227.2:4114 , piece_id=LVPMLQ5TTL7MWSU2DNQLMJ4HIBAPA67BCBG6TFQFOCCI2M6SUBSA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.820888231 (537h25m39.691971266s) [noise=true , address=145.79.227.36:4519 , piece_id=4SBVDEPHDYRIN5ZTYH4FEC2F3R264MLKBEHN7PCSEFLUEVFEOS5Q, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.821290857 (537h25m39.692373892s) [noise=true , address=77.90.31.13:4626 , piece_id=ELKATNPYENFMCBTRGCWY3BOT256YCZKQNT5RYKIF3HD5UOJ5W7XA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.821671392 (537h25m39.692754427s) [noise=true , address=145.79.227.49:4338 , piece_id=N73Y5XF77ATAJVTC7NVJUH7JY3YIDEGPBAPQTYAZKZ6VHQ3PQKAA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.823121984 (537h25m39.69420502s) [noise=true , address=77.90.31.10:4517 , piece_id=GYVV7NIQC42S33GIXBONCNXREWQYFDMU4JOKGJHEXS4YFEW7UBDA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.823437135 (537h25m39.69452018s) [noise=true , address=145.79.227.47:4265 , piece_id=VZVENKQZUZ2WBLT2YC3MQNFLXGOSP7HJXV4CJYH52HIS6BYD4JDA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.824628483 (537h25m39.695711528s) [noise=true , address=45.91.116.12:4227 , piece_id=JCJYX6VT3ZTBAAH3BFQKBVFML5QMSLANI7ROSS2AHUCS4A6BTKGQ, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.826070598 (537h25m39.697153644s) [noise=true , address=145.79.227.11:4729 , piece_id=QNKGXVVNJZ2H7WGE3H5N3SIJSLIGUKGR6VV6CSLGGFPZTCWE2N5Q, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.826550502 (537h25m39.697633548s) [noise=true , address=45.91.116.4:4940 , piece_id=KTXJEJ6EYN6ZTNEAS65XBDQ76V6SLDFTDMI4U5FLUQQYBLUK3LGA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.828033787 (537h25m39.699116822s) [noise=true , address=77.90.31.19:4841 , piece_id=GL4D6YICDUV3TOIMJP2V6XOGIQ5L6M52I7JZEVNHF4KIXQXYS7WA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.829612443 (537h25m39.700695478s) [noise=true , address=145.79.227.37:4566 , piece_id=SWNPU6GQF7OV3AH5QPBVVPPRUKTUJH2NEKAQ6BN3GQQ3XGDABWGQ, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:41.535470148 (537h25m41.406553183s) [upload=471336]: Next returned nil segment Dec 28 00:54:41.535483433 (537h25m41.406566468s) [upload=471336]: Waiting for error group managing segments... Dec 28 00:54:41.825793310 (537h25m41.696876355s) [noise=true , address=145.79.227.2:4114 , piece_id=LVPMLQ5TTL7MWSU2DNQLMJ4HIBAPA67BCBG6TFQFOCCI2M6SUBSA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.839695201 (537h25m41.710778246s) [noise=true , address=145.79.227.47:4265 , piece_id=VZVENKQZUZ2WBLT2YC3MQNFLXGOSP7HJXV4CJYH52HIS6BYD4JDA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.840397630 (537h25m41.711480695s) [noise=true , address=145.79.227.36:4519 , piece_id=4SBVDEPHDYRIN5ZTYH4FEC2F3R264MLKBEHN7PCSEFLUEVFEOS5Q, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.842129898 (537h25m41.713212933s) [noise=true , address=145.79.227.19:4077 , piece_id=LOVU5N3YZLT5R2EZ37STPIIWWQD4B6PT3XHNOEK6LPGMSVZPRNFQ, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.848645064 (537h25m41.719728099s) [noise=true , address=145.79.227.37:4566 , piece_id=SWNPU6GQF7OV3AH5QPBVVPPRUKTUJH2NEKAQ6BN3GQQ3XGDABWGQ, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.854569996 (537h25m41.725653041s) [noise=true , address=145.79.227.11:4729 , piece_id=QNKGXVVNJZ2H7WGE3H5N3SIJSLIGUKGR6VV6CSLGGFPZTCWE2N5Q, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.865492594 (537h25m41.736575629s) [noise=true , address=145.79.227.42:4122 , piece_id=4WUUPI5HFJVNLZWQVGJ6PEP3HOZ42Z5FXRKUTZWUJT3UIZFEJ5LA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.010495543 (537h25m41.881578588s) [noise=true , address=77.90.31.10:4517 , piece_id=GYVV7NIQC42S33GIXBONCNXREWQYFDMU4JOKGJHEXS4YFEW7UBDA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.040443720 (537h25m41.911526765s) [noise=true , address=77.90.31.19:4841 , piece_id=GL4D6YICDUV3TOIMJP2V6XOGIQ5L6M52I7JZEVNHF4KIXQXYS7WA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.045037658 (537h25m41.916120703s) [noise=true , address=77.90.31.13:4626 , piece_id=ELKATNPYENFMCBTRGCWY3BOT256YCZKQNT5RYKIF3HD5UOJ5W7XA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.048156847 (537h25m41.919239882s) [noise=true , address=77.90.31.12:4590 , piece_id=Y4OAPBHUU4OMR5BK5UMI7KMXSPKFMNC36ESCVIAHRU7LUSM6BCFA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.117720708 (537h25m41.988803753s) [noise=true , address=45.91.116.12:4227 , piece_id=JCJYX6VT3ZTBAAH3BFQKBVFML5QMSLANI7ROSS2AHUCS4A6BTKGQ, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.119361613 (537h25m41.990444658s) [noise=true , address=45.91.116.7:4011 , piece_id=OQWCFH43GXQSXQOUWZ66OFOH5FLEPKIMBZOPYN5FOEBZMMDLNUBA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.119374126 (537h25m41.990457161s) [seg_pos={0 0}, upload=471336]: Segment reached optimal threshold of 13 pieces. Dec 28 00:54:42.119578986 (537h25m41.990662021s) [noise=true , address=145.79.227.48:4301 , piece_id=GVYJFKDHEIYL6JCGOENR3QMPVHZEOQTGD7SHKJNLIDIPFD4YIPQA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29): protocol: expected piece hash; context canceled; EOF Dec 28 00:54:42.119588113 (537h25m41.990671158s) [noise=true , address=145.79.227.48:4301 , piece_id=GVYJFKDHEIYL6JCGOENR3QMPVHZEOQTGD7SHKJNLIDIPFD4YIPQA, seg_pos={0 0}, upload=471336]: Long tail cancellation due to optimal threshold Dec 28 00:54:42.119598444 (537h25m41.990681479s) [noise=true , address=45.91.116.4:4940 , piece_id=KTXJEJ6EYN6ZTNEAS65XBDQ76V6SLDFTDMI4U5FLUQQYBLUK3LGA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 28 00:54:42.119608703 (537h25m41.990691748s) [noise=true , address=45.91.116.4:4940 , piece_id=KTXJEJ6EYN6ZTNEAS65XBDQ76V6SLDFTDMI4U5FLUQQYBLUK3LGA, seg_pos={0 0}, upload=471336]: Long tail cancellation due to optimal threshold Dec 28 00:54:42.119752948 (537h25m41.990835993s) [noise=true , address=145.79.227.49:4338 , piece_id=N73Y5XF77ATAJVTC7NVJUH7JY3YIDEGPBAPQTYAZKZ6VHQ3PQKAA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 28 00:54:42.119761714 (537h25m41.990844749s) [noise=true , address=145.79.227.49:4338 , piece_id=N73Y5XF77ATAJVTC7NVJUH7JY3YIDEGPBAPQTYAZKZ6VHQ3PQKAA, seg_pos={0 0}, upload=471336]: Long tail cancellation due to optimal threshold Dec 28 00:54:42.119784458 (537h25m41.990867503s) [seg_pos={0 0}, upload=471336]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:54:42.119881543 (537h25m41.990964588s) [upload=471336]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:54:42.119919064 (537h25m41.991002109s) [upload=471336]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:54:42.419014862 (537h25m42.290097907s) [upload=471336]: Done uploading segments... Dec 28 00:54:42.419031904 (537h25m42.290114939s) [upload=471336]: Upload finished. err: <nil> Duration: 5.967777971s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 24.984547463s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 00:56:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 24.984574986s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 31.414472817s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 00:56:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 31.414500752s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [.XXX.X.X.XX.X.XXX...X.X.XX.XXX....X...X.X.X...XXXXX.X.XXX.X.] duration: 13.054402846s trace: https://jaeger.dev.storj.io/trace/43b8ae0fc43abb1d FAIL (2025-12-28 00:55:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 00:54:39.571036892 (537h25m39.442119937s) [upload=471336]: Starting upload Dec 28 00:54:39.571112146 (537h25m39.442195181s) [upload=471336]: Done starting upload Dec 28 00:54:39.571146471 (537h25m39.442229516s) [upload=471336]: Uploading segments... Dec 28 00:54:39.571352564 (537h25m39.442435609s) [upload=471336]: Got next segment. Inline: false Dec 28 00:54:39.571389333 (537h25m39.442472379s) [upload=471336]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:54:39.571394974 (537h25m39.442478019s) [upload=471336]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:54:39.819105876 (537h25m39.690188922s) [seg_pos={0 0}, upload=471336]: Begin upload segment... Dec 28 00:54:39.819160561 (537h25m39.690243596s) [seg_pos={0 0}, upload=471336]: Done begin upload segment. Dec 28 00:54:39.819212550 (537h25m39.690295585s) [noise=true , address=145.79.227.42:4122 , piece_id=4WUUPI5HFJVNLZWQVGJ6PEP3HOZ42Z5FXRKUTZWUJT3UIZFEJ5LA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.819219323 (537h25m39.690302358s) [noise=true , address=77.90.31.12:4590 , piece_id=Y4OAPBHUU4OMR5BK5UMI7KMXSPKFMNC36ESCVIAHRU7LUSM6BCFA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.819792885 (537h25m39.69087592s) [noise=true , address=45.91.116.7:4011 , piece_id=OQWCFH43GXQSXQOUWZ66OFOH5FLEPKIMBZOPYN5FOEBZMMDLNUBA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.820079420 (537h25m39.691162455s) [noise=true , address=145.79.227.48:4301 , piece_id=GVYJFKDHEIYL6JCGOENR3QMPVHZEOQTGD7SHKJNLIDIPFD4YIPQA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.820343273 (537h25m39.691426308s) [noise=true , address=145.79.227.19:4077 , piece_id=LOVU5N3YZLT5R2EZ37STPIIWWQD4B6PT3XHNOEK6LPGMSVZPRNFQ, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.820600693 (537h25m39.691683728s) [noise=true , address=145.79.227.2:4114 , piece_id=LVPMLQ5TTL7MWSU2DNQLMJ4HIBAPA67BCBG6TFQFOCCI2M6SUBSA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.820888231 (537h25m39.691971266s) [noise=true , address=145.79.227.36:4519 , piece_id=4SBVDEPHDYRIN5ZTYH4FEC2F3R264MLKBEHN7PCSEFLUEVFEOS5Q, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.821290857 (537h25m39.692373892s) [noise=true , address=77.90.31.13:4626 , piece_id=ELKATNPYENFMCBTRGCWY3BOT256YCZKQNT5RYKIF3HD5UOJ5W7XA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.821671392 (537h25m39.692754427s) [noise=true , address=145.79.227.49:4338 , piece_id=N73Y5XF77ATAJVTC7NVJUH7JY3YIDEGPBAPQTYAZKZ6VHQ3PQKAA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.823121984 (537h25m39.69420502s) [noise=true , address=77.90.31.10:4517 , piece_id=GYVV7NIQC42S33GIXBONCNXREWQYFDMU4JOKGJHEXS4YFEW7UBDA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.823437135 (537h25m39.69452018s) [noise=true , address=145.79.227.47:4265 , piece_id=VZVENKQZUZ2WBLT2YC3MQNFLXGOSP7HJXV4CJYH52HIS6BYD4JDA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.824628483 (537h25m39.695711528s) [noise=true , address=45.91.116.12:4227 , piece_id=JCJYX6VT3ZTBAAH3BFQKBVFML5QMSLANI7ROSS2AHUCS4A6BTKGQ, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.826070598 (537h25m39.697153644s) [noise=true , address=145.79.227.11:4729 , piece_id=QNKGXVVNJZ2H7WGE3H5N3SIJSLIGUKGR6VV6CSLGGFPZTCWE2N5Q, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.826550502 (537h25m39.697633548s) [noise=true , address=45.91.116.4:4940 , piece_id=KTXJEJ6EYN6ZTNEAS65XBDQ76V6SLDFTDMI4U5FLUQQYBLUK3LGA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.828033787 (537h25m39.699116822s) [noise=true , address=77.90.31.19:4841 , piece_id=GL4D6YICDUV3TOIMJP2V6XOGIQ5L6M52I7JZEVNHF4KIXQXYS7WA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.829612443 (537h25m39.700695478s) [noise=true , address=145.79.227.37:4566 , piece_id=SWNPU6GQF7OV3AH5QPBVVPPRUKTUJH2NEKAQ6BN3GQQ3XGDABWGQ, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:41.535470148 (537h25m41.406553183s) [upload=471336]: Next returned nil segment Dec 28 00:54:41.535483433 (537h25m41.406566468s) [upload=471336]: Waiting for error group managing segments... Dec 28 00:54:41.825793310 (537h25m41.696876355s) [noise=true , address=145.79.227.2:4114 , piece_id=LVPMLQ5TTL7MWSU2DNQLMJ4HIBAPA67BCBG6TFQFOCCI2M6SUBSA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.839695201 (537h25m41.710778246s) [noise=true , address=145.79.227.47:4265 , piece_id=VZVENKQZUZ2WBLT2YC3MQNFLXGOSP7HJXV4CJYH52HIS6BYD4JDA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.840397630 (537h25m41.711480695s) [noise=true , address=145.79.227.36:4519 , piece_id=4SBVDEPHDYRIN5ZTYH4FEC2F3R264MLKBEHN7PCSEFLUEVFEOS5Q, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.842129898 (537h25m41.713212933s) [noise=true , address=145.79.227.19:4077 , piece_id=LOVU5N3YZLT5R2EZ37STPIIWWQD4B6PT3XHNOEK6LPGMSVZPRNFQ, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.848645064 (537h25m41.719728099s) [noise=true , address=145.79.227.37:4566 , piece_id=SWNPU6GQF7OV3AH5QPBVVPPRUKTUJH2NEKAQ6BN3GQQ3XGDABWGQ, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.854569996 (537h25m41.725653041s) [noise=true , address=145.79.227.11:4729 , piece_id=QNKGXVVNJZ2H7WGE3H5N3SIJSLIGUKGR6VV6CSLGGFPZTCWE2N5Q, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.865492594 (537h25m41.736575629s) [noise=true , address=145.79.227.42:4122 , piece_id=4WUUPI5HFJVNLZWQVGJ6PEP3HOZ42Z5FXRKUTZWUJT3UIZFEJ5LA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.010495543 (537h25m41.881578588s) [noise=true , address=77.90.31.10:4517 , piece_id=GYVV7NIQC42S33GIXBONCNXREWQYFDMU4JOKGJHEXS4YFEW7UBDA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.040443720 (537h25m41.911526765s) [noise=true , address=77.90.31.19:4841 , piece_id=GL4D6YICDUV3TOIMJP2V6XOGIQ5L6M52I7JZEVNHF4KIXQXYS7WA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.045037658 (537h25m41.916120703s) [noise=true , address=77.90.31.13:4626 , piece_id=ELKATNPYENFMCBTRGCWY3BOT256YCZKQNT5RYKIF3HD5UOJ5W7XA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.048156847 (537h25m41.919239882s) [noise=true , address=77.90.31.12:4590 , piece_id=Y4OAPBHUU4OMR5BK5UMI7KMXSPKFMNC36ESCVIAHRU7LUSM6BCFA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.117720708 (537h25m41.988803753s) [noise=true , address=45.91.116.12:4227 , piece_id=JCJYX6VT3ZTBAAH3BFQKBVFML5QMSLANI7ROSS2AHUCS4A6BTKGQ, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.119361613 (537h25m41.990444658s) [noise=true , address=45.91.116.7:4011 , piece_id=OQWCFH43GXQSXQOUWZ66OFOH5FLEPKIMBZOPYN5FOEBZMMDLNUBA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.119374126 (537h25m41.990457161s) [seg_pos={0 0}, upload=471336]: Segment reached optimal threshold of 13 pieces. Dec 28 00:54:42.119578986 (537h25m41.990662021s) [noise=true , address=145.79.227.48:4301 , piece_id=GVYJFKDHEIYL6JCGOENR3QMPVHZEOQTGD7SHKJNLIDIPFD4YIPQA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29): protocol: expected piece hash; context canceled; EOF Dec 28 00:54:42.119588113 (537h25m41.990671158s) [noise=true , address=145.79.227.48:4301 , piece_id=GVYJFKDHEIYL6JCGOENR3QMPVHZEOQTGD7SHKJNLIDIPFD4YIPQA, seg_pos={0 0}, upload=471336]: Long tail cancellation due to optimal threshold Dec 28 00:54:42.119598444 (537h25m41.990681479s) [noise=true , address=45.91.116.4:4940 , piece_id=KTXJEJ6EYN6ZTNEAS65XBDQ76V6SLDFTDMI4U5FLUQQYBLUK3LGA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 28 00:54:42.119608703 (537h25m41.990691748s) [noise=true , address=45.91.116.4:4940 , piece_id=KTXJEJ6EYN6ZTNEAS65XBDQ76V6SLDFTDMI4U5FLUQQYBLUK3LGA, seg_pos={0 0}, upload=471336]: Long tail cancellation due to optimal threshold Dec 28 00:54:42.119752948 (537h25m41.990835993s) [noise=true , address=145.79.227.49:4338 , piece_id=N73Y5XF77ATAJVTC7NVJUH7JY3YIDEGPBAPQTYAZKZ6VHQ3PQKAA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 28 00:54:42.119761714 (537h25m41.990844749s) [noise=true , address=145.79.227.49:4338 , piece_id=N73Y5XF77ATAJVTC7NVJUH7JY3YIDEGPBAPQTYAZKZ6VHQ3PQKAA, seg_pos={0 0}, upload=471336]: Long tail cancellation due to optimal threshold Dec 28 00:54:42.119784458 (537h25m41.990867503s) [seg_pos={0 0}, upload=471336]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:54:42.119881543 (537h25m41.990964588s) [upload=471336]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:54:42.119919064 (537h25m41.991002109s) [upload=471336]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:54:42.419014862 (537h25m42.290097907s) [upload=471336]: Done uploading segments... Dec 28 00:54:42.419031904 (537h25m42.290114939s) [upload=471336]: Upload finished. err: <nil> Duration: 13.054440638s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 32.339611964s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 00:55:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 32.339628304s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [X.XXX.X.X.XX.X.XXX...X.X.XX.XXX....X...X.X.X...XXXXX.X.XXX.X] duration: 30.003653216s trace: https://jaeger.dev.storj.io/trace/43b8ae0fc43abb1d FAIL (2025-12-28 00:55:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 00:54:39.571036892 (537h25m39.442119937s) [upload=471336]: Starting upload Dec 28 00:54:39.571112146 (537h25m39.442195181s) [upload=471336]: Done starting upload Dec 28 00:54:39.571146471 (537h25m39.442229516s) [upload=471336]: Uploading segments... Dec 28 00:54:39.571352564 (537h25m39.442435609s) [upload=471336]: Got next segment. Inline: false Dec 28 00:54:39.571389333 (537h25m39.442472379s) [upload=471336]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:54:39.571394974 (537h25m39.442478019s) [upload=471336]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:54:39.819105876 (537h25m39.690188922s) [seg_pos={0 0}, upload=471336]: Begin upload segment... Dec 28 00:54:39.819160561 (537h25m39.690243596s) [seg_pos={0 0}, upload=471336]: Done begin upload segment. Dec 28 00:54:39.819212550 (537h25m39.690295585s) [noise=true , address=145.79.227.42:4122 , piece_id=4WUUPI5HFJVNLZWQVGJ6PEP3HOZ42Z5FXRKUTZWUJT3UIZFEJ5LA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.819219323 (537h25m39.690302358s) [noise=true , address=77.90.31.12:4590 , piece_id=Y4OAPBHUU4OMR5BK5UMI7KMXSPKFMNC36ESCVIAHRU7LUSM6BCFA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.819792885 (537h25m39.69087592s) [noise=true , address=45.91.116.7:4011 , piece_id=OQWCFH43GXQSXQOUWZ66OFOH5FLEPKIMBZOPYN5FOEBZMMDLNUBA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.820079420 (537h25m39.691162455s) [noise=true , address=145.79.227.48:4301 , piece_id=GVYJFKDHEIYL6JCGOENR3QMPVHZEOQTGD7SHKJNLIDIPFD4YIPQA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.820343273 (537h25m39.691426308s) [noise=true , address=145.79.227.19:4077 , piece_id=LOVU5N3YZLT5R2EZ37STPIIWWQD4B6PT3XHNOEK6LPGMSVZPRNFQ, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.820600693 (537h25m39.691683728s) [noise=true , address=145.79.227.2:4114 , piece_id=LVPMLQ5TTL7MWSU2DNQLMJ4HIBAPA67BCBG6TFQFOCCI2M6SUBSA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.820888231 (537h25m39.691971266s) [noise=true , address=145.79.227.36:4519 , piece_id=4SBVDEPHDYRIN5ZTYH4FEC2F3R264MLKBEHN7PCSEFLUEVFEOS5Q, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.821290857 (537h25m39.692373892s) [noise=true , address=77.90.31.13:4626 , piece_id=ELKATNPYENFMCBTRGCWY3BOT256YCZKQNT5RYKIF3HD5UOJ5W7XA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.821671392 (537h25m39.692754427s) [noise=true , address=145.79.227.49:4338 , piece_id=N73Y5XF77ATAJVTC7NVJUH7JY3YIDEGPBAPQTYAZKZ6VHQ3PQKAA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.823121984 (537h25m39.69420502s) [noise=true , address=77.90.31.10:4517 , piece_id=GYVV7NIQC42S33GIXBONCNXREWQYFDMU4JOKGJHEXS4YFEW7UBDA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.823437135 (537h25m39.69452018s) [noise=true , address=145.79.227.47:4265 , piece_id=VZVENKQZUZ2WBLT2YC3MQNFLXGOSP7HJXV4CJYH52HIS6BYD4JDA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.824628483 (537h25m39.695711528s) [noise=true , address=45.91.116.12:4227 , piece_id=JCJYX6VT3ZTBAAH3BFQKBVFML5QMSLANI7ROSS2AHUCS4A6BTKGQ, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.826070598 (537h25m39.697153644s) [noise=true , address=145.79.227.11:4729 , piece_id=QNKGXVVNJZ2H7WGE3H5N3SIJSLIGUKGR6VV6CSLGGFPZTCWE2N5Q, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.826550502 (537h25m39.697633548s) [noise=true , address=45.91.116.4:4940 , piece_id=KTXJEJ6EYN6ZTNEAS65XBDQ76V6SLDFTDMI4U5FLUQQYBLUK3LGA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.828033787 (537h25m39.699116822s) [noise=true , address=77.90.31.19:4841 , piece_id=GL4D6YICDUV3TOIMJP2V6XOGIQ5L6M52I7JZEVNHF4KIXQXYS7WA, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:39.829612443 (537h25m39.700695478s) [noise=true , address=145.79.227.37:4566 , piece_id=SWNPU6GQF7OV3AH5QPBVVPPRUKTUJH2NEKAQ6BN3GQQ3XGDABWGQ, seg_pos={0 0}, upload=471336]: Uploading piece... Dec 28 00:54:41.535470148 (537h25m41.406553183s) [upload=471336]: Next returned nil segment Dec 28 00:54:41.535483433 (537h25m41.406566468s) [upload=471336]: Waiting for error group managing segments... Dec 28 00:54:41.825793310 (537h25m41.696876355s) [noise=true , address=145.79.227.2:4114 , piece_id=LVPMLQ5TTL7MWSU2DNQLMJ4HIBAPA67BCBG6TFQFOCCI2M6SUBSA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.839695201 (537h25m41.710778246s) [noise=true , address=145.79.227.47:4265 , piece_id=VZVENKQZUZ2WBLT2YC3MQNFLXGOSP7HJXV4CJYH52HIS6BYD4JDA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.840397630 (537h25m41.711480695s) [noise=true , address=145.79.227.36:4519 , piece_id=4SBVDEPHDYRIN5ZTYH4FEC2F3R264MLKBEHN7PCSEFLUEVFEOS5Q, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.842129898 (537h25m41.713212933s) [noise=true , address=145.79.227.19:4077 , piece_id=LOVU5N3YZLT5R2EZ37STPIIWWQD4B6PT3XHNOEK6LPGMSVZPRNFQ, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.848645064 (537h25m41.719728099s) [noise=true , address=145.79.227.37:4566 , piece_id=SWNPU6GQF7OV3AH5QPBVVPPRUKTUJH2NEKAQ6BN3GQQ3XGDABWGQ, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.854569996 (537h25m41.725653041s) [noise=true , address=145.79.227.11:4729 , piece_id=QNKGXVVNJZ2H7WGE3H5N3SIJSLIGUKGR6VV6CSLGGFPZTCWE2N5Q, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:41.865492594 (537h25m41.736575629s) [noise=true , address=145.79.227.42:4122 , piece_id=4WUUPI5HFJVNLZWQVGJ6PEP3HOZ42Z5FXRKUTZWUJT3UIZFEJ5LA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.010495543 (537h25m41.881578588s) [noise=true , address=77.90.31.10:4517 , piece_id=GYVV7NIQC42S33GIXBONCNXREWQYFDMU4JOKGJHEXS4YFEW7UBDA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.040443720 (537h25m41.911526765s) [noise=true , address=77.90.31.19:4841 , piece_id=GL4D6YICDUV3TOIMJP2V6XOGIQ5L6M52I7JZEVNHF4KIXQXYS7WA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.045037658 (537h25m41.916120703s) [noise=true , address=77.90.31.13:4626 , piece_id=ELKATNPYENFMCBTRGCWY3BOT256YCZKQNT5RYKIF3HD5UOJ5W7XA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.048156847 (537h25m41.919239882s) [noise=true , address=77.90.31.12:4590 , piece_id=Y4OAPBHUU4OMR5BK5UMI7KMXSPKFMNC36ESCVIAHRU7LUSM6BCFA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.117720708 (537h25m41.988803753s) [noise=true , address=45.91.116.12:4227 , piece_id=JCJYX6VT3ZTBAAH3BFQKBVFML5QMSLANI7ROSS2AHUCS4A6BTKGQ, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.119361613 (537h25m41.990444658s) [noise=true , address=45.91.116.7:4011 , piece_id=OQWCFH43GXQSXQOUWZ66OFOH5FLEPKIMBZOPYN5FOEBZMMDLNUBA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: <nil> Dec 28 00:54:42.119374126 (537h25m41.990457161s) [seg_pos={0 0}, upload=471336]: Segment reached optimal threshold of 13 pieces. Dec 28 00:54:42.119578986 (537h25m41.990662021s) [noise=true , address=145.79.227.48:4301 , piece_id=GVYJFKDHEIYL6JCGOENR3QMPVHZEOQTGD7SHKJNLIDIPFD4YIPQA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29): protocol: expected piece hash; context canceled; EOF Dec 28 00:54:42.119588113 (537h25m41.990671158s) [noise=true , address=145.79.227.48:4301 , piece_id=GVYJFKDHEIYL6JCGOENR3QMPVHZEOQTGD7SHKJNLIDIPFD4YIPQA, seg_pos={0 0}, upload=471336]: Long tail cancellation due to optimal threshold Dec 28 00:54:42.119598444 (537h25m41.990681479s) [noise=true , address=45.91.116.4:4940 , piece_id=KTXJEJ6EYN6ZTNEAS65XBDQ76V6SLDFTDMI4U5FLUQQYBLUK3LGA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 28 00:54:42.119608703 (537h25m41.990691748s) [noise=true , address=45.91.116.4:4940 , piece_id=KTXJEJ6EYN6ZTNEAS65XBDQ76V6SLDFTDMI4U5FLUQQYBLUK3LGA, seg_pos={0 0}, upload=471336]: Long tail cancellation due to optimal threshold Dec 28 00:54:42.119752948 (537h25m41.990835993s) [noise=true , address=145.79.227.49:4338 , piece_id=N73Y5XF77ATAJVTC7NVJUH7JY3YIDEGPBAPQTYAZKZ6VHQ3PQKAA, seg_pos={0 0}, upload=471336]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF Dec 28 00:54:42.119761714 (537h25m41.990844749s) [noise=true , address=145.79.227.49:4338 , piece_id=N73Y5XF77ATAJVTC7NVJUH7JY3YIDEGPBAPQTYAZKZ6VHQ3PQKAA, seg_pos={0 0}, upload=471336]: Long tail cancellation due to optimal threshold Dec 28 00:54:42.119784458 (537h25m41.990867503s) [seg_pos={0 0}, upload=471336]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:54:42.119881543 (537h25m41.990964588s) [upload=471336]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:54:42.119919064 (537h25m41.991002109s) [upload=471336]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:54:42.419014862 (537h25m42.290097907s) [upload=471336]: Done uploading segments... Dec 28 00:54:42.419031904 (537h25m42.290114939s) [upload=471336]: Upload finished. err: <nil> Duration: 30.003680808s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 22.526216332s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 00:54:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 22.526245176s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 36.818949848s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 00:54:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 36.818979235s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [.X.XXX.X.X.XX.X.XXX...X.X.XX.XXX....X...X.X.X...XXXXX.X.XXX.] duration: 6.252786764s trace: https://jaeger.dev.storj.io/trace/a15d2188dd84399 FAIL (2025-12-28 00:54:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: Z2AVU4HRZBPJTRTPAWVULCSWKZTRY2MSYOANU7MUB27QDJEMMEZA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: ZOYAJNG2NSUWQRDKBIFWSGG77BK524FL4T3JDNXLCOUAZZ35NNNA) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: HDLYXKJXAUEALNHDW35ME4RH6YZGRHLCMHZ3ZHJCTPI5NPZLA2CA) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: FNOE2CFFAKKWMYY2KSQECFVO6Y36JHE7CJZ7UVTYB7LIOAIIZDJQ) context deadline exceeded ======LOGS====== 8 00:53:17.596126537 (537h24m17.467209582s) [upload=471314]: Starting upload Dec 28 00:53:17.596180970 (537h24m17.467264015s) [upload=471314]: Done starting upload Dec 28 00:53:17.596280680 (537h24m17.467363715s) [upload=471314]: Uploading segments... Dec 28 00:53:17.596460573 (537h24m17.467543618s) [upload=471314]: Got next segment. Inline: false Dec 28 00:53:17.596530095 (537h24m17.46761314s) [upload=471314]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:53:17.596537921 (537h24m17.467620966s) [upload=471314]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:53:17.839703236 (537h24m17.710786281s) [seg_pos={0 0}, upload=471314]: Begin upload segment... Dec 28 00:53:17.839754034 (537h24m17.710837069s) [seg_pos={0 0}, upload=471314]: Done begin upload segment. Dec 28 00:53:17.839771627 (537h24m17.710854662s) [noise=true , address=145.79.227.36:4518 , piece_id=DMCMIDRIAHSWNFYFJHVWQENWA7BBJLYP4PR3INHPHMP6X7UUQ53Q, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.839780484 (537h24m17.710863519s) [noise=true , address=145.79.227.24:4583 , piece_id=ZOYAJNG2NSUWQRDKBIFWSGG77BK524FL4T3JDNXLCOUAZZ35NNNA, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.840133737 (537h24m17.711216782s) [noise=true , address=45.91.116.12:4226 , piece_id=MOFNZDLYQBXBBN7DDTF4T22K364CQDLS2DTICEPEGXYW4OXY47RA, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.840760450 (537h24m17.711843485s) [noise=true , address=145.79.227.4:4186 , piece_id=VUV3VHGS3RXOGUXCUF6FERGV7V6SGZVHOYUHJVDTWW4FR2V6EG2A, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.841064990 (537h24m17.712148025s) [noise=true , address=45.91.116.10:4081 , piece_id=ZNEJH54VTSM2OMBNETNJAM4D337YFEHVC5KE76AI3PNVG57DEGTA, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.841364650 (537h24m17.712447695s) [noise=true , address=45.91.116.8:4047 , piece_id=HDLYXKJXAUEALNHDW35ME4RH6YZGRHLCMHZ3ZHJCTPI5NPZLA2CA, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.841624686 (537h24m17.712707731s) [noise=true , address=145.79.227.39:4015 , piece_id=JCJKYVDAZTDP63JDC4R5X7Y5OMMCR56QEJAPCNAFVCNGGIMZU37A, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.841918625 (537h24m17.71300166s) [noise=true , address=77.90.31.16:4735 , piece_id=FNOE2CFFAKKWMYY2KSQECFVO6Y36JHE7CJZ7UVTYB7LIOAIIZDJQ, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.842222734 (537h24m17.713305769s) [noise=true , address=45.91.116.5:4299 , piece_id=Z2AVU4HRZBPJTRTPAWVULCSWKZTRY2MSYOANU7MUB27QDJEMMEZA, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.843130152 (537h24m17.714213187s) [noise=true , address=145.79.227.11:4731 , piece_id=QBEUKPCJ2R2TSHFBQXVQSPJQQJXROHBP55FAEN5WQYEV5GPU2E2Q, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.843293894 (537h24m17.714376929s) [noise=true , address=45.91.116.4:4939 , piece_id=HUCVIO45HEIJFGGLZANB23TMHKDRCGEHE3V7Q34C54IH353YSHOA, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.844176515 (537h24m17.71525955s) [noise=true , address=45.91.116.13:4153 , piece_id=7K7R5QY63QDZSFWLRK7ZKVGAK3RPABQTRVBQSI4G5BFPOV7HMOJQ, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.845030610 (537h24m17.716113656s) [noise=true , address=145.79.227.3:4150 , piece_id=5DN3ECBYRXSTFGG4RCPUF24LDKTCHBUTUHJAELSHOVBDR5MOMG4A, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.846051455 (537h24m17.71713449s) [noise=true , address=45.91.116.14:4189 , piece_id=3JUWNWFPMRYU7VF5WD4IWUM5PNGLXITYIYDVOMDZSFYLWMQ6ZSPA, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.847369986 (537h24m17.718453021s) [noise=true , address=145.79.227.19:4079 , piece_id=I3UO3747YY7EBKVDFISXM3N5DPA42BEBFXMHSQXAY4ZP2M2UXCMA, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.849424648 (537h24m17.720507683s) [noise=true , address=45.91.116.7:4010 , piece_id=JHSUSJMDKVHUSII4D34MYSLRNZFDF5BLBKWIFC6LX3KKZRCSWKLQ, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:19.453212734 (537h24m19.324295769s) [upload=471314]: Next returned nil segment Dec 28 00:53:19.453225879 (537h24m19.324308914s) [upload=471314]: Waiting for error group managing segments... Dec 28 00:53:19.836508783 (537h24m19.707591818s) [noise=true , address=145.79.227.19:4079 , piece_id=I3UO3747YY7EBKVDFISXM3N5DPA42BEBFXMHSQXAY4ZP2M2UXCMA, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:19.839183255 (537h24m19.71026629s) [noise=true , address=145.79.227.24:4583 , piece_id=ZOYAJNG2NSUWQRDKBIFWSGG77BK524FL4T3JDNXLCOUAZZ35NNNA, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:19.840850039 (537h24m19.711933084s) [noise=true , address=145.79.227.11:4731 , piece_id=QBEUKPCJ2R2TSHFBQXVQSPJQQJXROHBP55FAEN5WQYEV5GPU2E2Q, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:19.848938079 (537h24m19.720021115s) [noise=true , address=145.79.227.4:4186 , piece_id=VUV3VHGS3RXOGUXCUF6FERGV7V6SGZVHOYUHJVDTWW4FR2V6EG2A, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:19.851779490 (537h24m19.722862525s) [noise=true , address=145.79.227.3:4150 , piece_id=5DN3ECBYRXSTFGG4RCPUF24LDKTCHBUTUHJAELSHOVBDR5MOMG4A, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:19.861869403 (537h24m19.732952439s) [noise=true , address=145.79.227.36:4518 , piece_id=DMCMIDRIAHSWNFYFJHVWQENWA7BBJLYP4PR3INHPHMP6X7UUQ53Q, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:19.887942351 (537h24m19.759025386s) [noise=true , address=145.79.227.39:4015 , piece_id=JCJKYVDAZTDP63JDC4R5X7Y5OMMCR56QEJAPCNAFVCNGGIMZU37A, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:20.022520831 (537h24m19.893603866s) [noise=true , address=77.90.31.16:4735 , piece_id=FNOE2CFFAKKWMYY2KSQECFVO6Y36JHE7CJZ7UVTYB7LIOAIIZDJQ, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:20.088879838 (537h24m19.959962873s) [noise=true , address=45.91.116.4:4939 , piece_id=HUCVIO45HEIJFGGLZANB23TMHKDRCGEHE3V7Q34C54IH353YSHOA, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:20.109535119 (537h24m19.980618164s) [noise=true , address=45.91.116.5:4299 , piece_id=Z2AVU4HRZBPJTRTPAWVULCSWKZTRY2MSYOANU7MUB27QDJEMMEZA, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:20.150146541 (537h24m20.021229586s) [noise=true , address=45.91.116.8:4047 , piece_id=HDLYXKJXAUEALNHDW35ME4RH6YZGRHLCMHZ3ZHJCTPI5NPZLA2CA, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:20.172140200 (537h24m20.043223245s) [noise=true , address=45.91.116.13:4153 , piece_id=7K7R5QY63QDZSFWLRK7ZKVGAK3RPABQTRVBQSI4G5BFPOV7HMOJQ, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:20.177725925 (537h24m20.04880896s) [noise=true , address=45.91.116.7:4010 , piece_id=JHSUSJMDKVHUSII4D34MYSLRNZFDF5BLBKWIFC6LX3KKZRCSWKLQ, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:20.177829362 (537h24m20.048912397s) [seg_pos={0 0}, upload=471314]: Segment reached optimal threshold of 13 pieces. Dec 28 00:53:20.178008814 (537h24m20.049091849s) [noise=true , address=45.91.116.14:4189 , piece_id=3JUWNWFPMRYU7VF5WD4IWUM5PNGLXITYIYDVOMDZSFYLWMQ6ZSPA, seg_pos={0 0}, upload=471314]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:47846->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:47846->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47846->45.91.116.14:4189: use of closed network connection Dec 28 00:53:20.178015678 (537h24m20.049098733s) [noise=true , address=45.91.116.14:4189 , piece_id=3JUWNWFPMRYU7VF5WD4IWUM5PNGLXITYIYDVOMDZSFYLWMQ6ZSPA, seg_pos={0 0}, upload=471314]: Long tail cancellation due to optimal threshold Dec 28 00:53:20.178120217 (537h24m20.049203252s) [noise=true , address=45.91.116.10:4081 , piece_id=ZNEJH54VTSM2OMBNETNJAM4D337YFEHVC5KE76AI3PNVG57DEGTA, seg_pos={0 0}, upload=471314]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 28 00:53:20.178137339 (537h24m20.049220374s) [noise=true , address=45.91.116.10:4081 , piece_id=ZNEJH54VTSM2OMBNETNJAM4D337YFEHVC5KE76AI3PNVG57DEGTA, seg_pos={0 0}, upload=471314]: Long tail cancellation due to optimal threshold Dec 28 00:53:20.178174620 (537h24m20.049257655s) [noise=true , address=45.91.116.12:4226 , piece_id=MOFNZDLYQBXBBN7DDTF4T22K364CQDLS2DTICEPEGXYW4OXY47RA, seg_pos={0 0}, upload=471314]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 28 00:53:20.178185089 (537h24m20.049268124s) [noise=true , address=45.91.116.12:4226 , piece_id=MOFNZDLYQBXBBN7DDTF4T22K364CQDLS2DTICEPEGXYW4OXY47RA, seg_pos={0 0}, upload=471314]: Long tail cancellation due to optimal threshold Dec 28 00:53:20.178201261 (537h24m20.049284306s) [seg_pos={0 0}, upload=471314]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:53:20.178293467 (537h24m20.049376502s) [upload=471314]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:53:20.178299308 (537h24m20.049382343s) [upload=471314]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:53:20.466730187 (537h24m20.337813252s) [upload=471314]: Done uploading segments... Dec 28 00:53:20.466756076 (537h24m20.337839111s) [upload=471314]: Upload finished. err: <nil> Duration: 6.252811281s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [..X.XXX.X.X.XX.X.XXX...X.X.XX.XXX....X...X.X.X...XXXXX.X.XXX] duration: 30.002659318s trace: https://jaeger.dev.storj.io/trace/a15d2188dd84399 FAIL (2025-12-28 00:53:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: Z2AVU4HRZBPJTRTPAWVULCSWKZTRY2MSYOANU7MUB27QDJEMMEZA) context deadline exceeded; piecestore: (Node ID: 1MCX7LMdSk3iENRGS5tE33ViZbANXGu3Po8bpenu4FLNGRQX19, Piece ID: ZOYAJNG2NSUWQRDKBIFWSGG77BK524FL4T3JDNXLCOUAZZ35NNNA) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: HDLYXKJXAUEALNHDW35ME4RH6YZGRHLCMHZ3ZHJCTPI5NPZLA2CA) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: FNOE2CFFAKKWMYY2KSQECFVO6Y36JHE7CJZ7UVTYB7LIOAIIZDJQ) context deadline exceeded ======LOGS====== 8 00:53:17.596126537 (537h24m17.467209582s) [upload=471314]: Starting upload Dec 28 00:53:17.596180970 (537h24m17.467264015s) [upload=471314]: Done starting upload Dec 28 00:53:17.596280680 (537h24m17.467363715s) [upload=471314]: Uploading segments... Dec 28 00:53:17.596460573 (537h24m17.467543618s) [upload=471314]: Got next segment. Inline: false Dec 28 00:53:17.596530095 (537h24m17.46761314s) [upload=471314]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:53:17.596537921 (537h24m17.467620966s) [upload=471314]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:53:17.839703236 (537h24m17.710786281s) [seg_pos={0 0}, upload=471314]: Begin upload segment... Dec 28 00:53:17.839754034 (537h24m17.710837069s) [seg_pos={0 0}, upload=471314]: Done begin upload segment. Dec 28 00:53:17.839771627 (537h24m17.710854662s) [noise=true , address=145.79.227.36:4518 , piece_id=DMCMIDRIAHSWNFYFJHVWQENWA7BBJLYP4PR3INHPHMP6X7UUQ53Q, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.839780484 (537h24m17.710863519s) [noise=true , address=145.79.227.24:4583 , piece_id=ZOYAJNG2NSUWQRDKBIFWSGG77BK524FL4T3JDNXLCOUAZZ35NNNA, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.840133737 (537h24m17.711216782s) [noise=true , address=45.91.116.12:4226 , piece_id=MOFNZDLYQBXBBN7DDTF4T22K364CQDLS2DTICEPEGXYW4OXY47RA, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.840760450 (537h24m17.711843485s) [noise=true , address=145.79.227.4:4186 , piece_id=VUV3VHGS3RXOGUXCUF6FERGV7V6SGZVHOYUHJVDTWW4FR2V6EG2A, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.841064990 (537h24m17.712148025s) [noise=true , address=45.91.116.10:4081 , piece_id=ZNEJH54VTSM2OMBNETNJAM4D337YFEHVC5KE76AI3PNVG57DEGTA, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.841364650 (537h24m17.712447695s) [noise=true , address=45.91.116.8:4047 , piece_id=HDLYXKJXAUEALNHDW35ME4RH6YZGRHLCMHZ3ZHJCTPI5NPZLA2CA, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.841624686 (537h24m17.712707731s) [noise=true , address=145.79.227.39:4015 , piece_id=JCJKYVDAZTDP63JDC4R5X7Y5OMMCR56QEJAPCNAFVCNGGIMZU37A, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.841918625 (537h24m17.71300166s) [noise=true , address=77.90.31.16:4735 , piece_id=FNOE2CFFAKKWMYY2KSQECFVO6Y36JHE7CJZ7UVTYB7LIOAIIZDJQ, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.842222734 (537h24m17.713305769s) [noise=true , address=45.91.116.5:4299 , piece_id=Z2AVU4HRZBPJTRTPAWVULCSWKZTRY2MSYOANU7MUB27QDJEMMEZA, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.843130152 (537h24m17.714213187s) [noise=true , address=145.79.227.11:4731 , piece_id=QBEUKPCJ2R2TSHFBQXVQSPJQQJXROHBP55FAEN5WQYEV5GPU2E2Q, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.843293894 (537h24m17.714376929s) [noise=true , address=45.91.116.4:4939 , piece_id=HUCVIO45HEIJFGGLZANB23TMHKDRCGEHE3V7Q34C54IH353YSHOA, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.844176515 (537h24m17.71525955s) [noise=true , address=45.91.116.13:4153 , piece_id=7K7R5QY63QDZSFWLRK7ZKVGAK3RPABQTRVBQSI4G5BFPOV7HMOJQ, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.845030610 (537h24m17.716113656s) [noise=true , address=145.79.227.3:4150 , piece_id=5DN3ECBYRXSTFGG4RCPUF24LDKTCHBUTUHJAELSHOVBDR5MOMG4A, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.846051455 (537h24m17.71713449s) [noise=true , address=45.91.116.14:4189 , piece_id=3JUWNWFPMRYU7VF5WD4IWUM5PNGLXITYIYDVOMDZSFYLWMQ6ZSPA, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.847369986 (537h24m17.718453021s) [noise=true , address=145.79.227.19:4079 , piece_id=I3UO3747YY7EBKVDFISXM3N5DPA42BEBFXMHSQXAY4ZP2M2UXCMA, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:17.849424648 (537h24m17.720507683s) [noise=true , address=45.91.116.7:4010 , piece_id=JHSUSJMDKVHUSII4D34MYSLRNZFDF5BLBKWIFC6LX3KKZRCSWKLQ, seg_pos={0 0}, upload=471314]: Uploading piece... Dec 28 00:53:19.453212734 (537h24m19.324295769s) [upload=471314]: Next returned nil segment Dec 28 00:53:19.453225879 (537h24m19.324308914s) [upload=471314]: Waiting for error group managing segments... Dec 28 00:53:19.836508783 (537h24m19.707591818s) [noise=true , address=145.79.227.19:4079 , piece_id=I3UO3747YY7EBKVDFISXM3N5DPA42BEBFXMHSQXAY4ZP2M2UXCMA, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:19.839183255 (537h24m19.71026629s) [noise=true , address=145.79.227.24:4583 , piece_id=ZOYAJNG2NSUWQRDKBIFWSGG77BK524FL4T3JDNXLCOUAZZ35NNNA, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:19.840850039 (537h24m19.711933084s) [noise=true , address=145.79.227.11:4731 , piece_id=QBEUKPCJ2R2TSHFBQXVQSPJQQJXROHBP55FAEN5WQYEV5GPU2E2Q, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:19.848938079 (537h24m19.720021115s) [noise=true , address=145.79.227.4:4186 , piece_id=VUV3VHGS3RXOGUXCUF6FERGV7V6SGZVHOYUHJVDTWW4FR2V6EG2A, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:19.851779490 (537h24m19.722862525s) [noise=true , address=145.79.227.3:4150 , piece_id=5DN3ECBYRXSTFGG4RCPUF24LDKTCHBUTUHJAELSHOVBDR5MOMG4A, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:19.861869403 (537h24m19.732952439s) [noise=true , address=145.79.227.36:4518 , piece_id=DMCMIDRIAHSWNFYFJHVWQENWA7BBJLYP4PR3INHPHMP6X7UUQ53Q, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:19.887942351 (537h24m19.759025386s) [noise=true , address=145.79.227.39:4015 , piece_id=JCJKYVDAZTDP63JDC4R5X7Y5OMMCR56QEJAPCNAFVCNGGIMZU37A, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:20.022520831 (537h24m19.893603866s) [noise=true , address=77.90.31.16:4735 , piece_id=FNOE2CFFAKKWMYY2KSQECFVO6Y36JHE7CJZ7UVTYB7LIOAIIZDJQ, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:20.088879838 (537h24m19.959962873s) [noise=true , address=45.91.116.4:4939 , piece_id=HUCVIO45HEIJFGGLZANB23TMHKDRCGEHE3V7Q34C54IH353YSHOA, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:20.109535119 (537h24m19.980618164s) [noise=true , address=45.91.116.5:4299 , piece_id=Z2AVU4HRZBPJTRTPAWVULCSWKZTRY2MSYOANU7MUB27QDJEMMEZA, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:20.150146541 (537h24m20.021229586s) [noise=true , address=45.91.116.8:4047 , piece_id=HDLYXKJXAUEALNHDW35ME4RH6YZGRHLCMHZ3ZHJCTPI5NPZLA2CA, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:20.172140200 (537h24m20.043223245s) [noise=true , address=45.91.116.13:4153 , piece_id=7K7R5QY63QDZSFWLRK7ZKVGAK3RPABQTRVBQSI4G5BFPOV7HMOJQ, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:20.177725925 (537h24m20.04880896s) [noise=true , address=45.91.116.7:4010 , piece_id=JHSUSJMDKVHUSII4D34MYSLRNZFDF5BLBKWIFC6LX3KKZRCSWKLQ, seg_pos={0 0}, upload=471314]: Done uploading piece. err: <nil> Dec 28 00:53:20.177829362 (537h24m20.048912397s) [seg_pos={0 0}, upload=471314]: Segment reached optimal threshold of 13 pieces. Dec 28 00:53:20.178008814 (537h24m20.049091849s) [noise=true , address=45.91.116.14:4189 , piece_id=3JUWNWFPMRYU7VF5WD4IWUM5PNGLXITYIYDVOMDZSFYLWMQ6ZSPA, seg_pos={0 0}, upload=471314]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:47846->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:47846->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47846->45.91.116.14:4189: use of closed network connection Dec 28 00:53:20.178015678 (537h24m20.049098733s) [noise=true , address=45.91.116.14:4189 , piece_id=3JUWNWFPMRYU7VF5WD4IWUM5PNGLXITYIYDVOMDZSFYLWMQ6ZSPA, seg_pos={0 0}, upload=471314]: Long tail cancellation due to optimal threshold Dec 28 00:53:20.178120217 (537h24m20.049203252s) [noise=true , address=45.91.116.10:4081 , piece_id=ZNEJH54VTSM2OMBNETNJAM4D337YFEHVC5KE76AI3PNVG57DEGTA, seg_pos={0 0}, upload=471314]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 28 00:53:20.178137339 (537h24m20.049220374s) [noise=true , address=45.91.116.10:4081 , piece_id=ZNEJH54VTSM2OMBNETNJAM4D337YFEHVC5KE76AI3PNVG57DEGTA, seg_pos={0 0}, upload=471314]: Long tail cancellation due to optimal threshold Dec 28 00:53:20.178174620 (537h24m20.049257655s) [noise=true , address=45.91.116.12:4226 , piece_id=MOFNZDLYQBXBBN7DDTF4T22K364CQDLS2DTICEPEGXYW4OXY47RA, seg_pos={0 0}, upload=471314]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 28 00:53:20.178185089 (537h24m20.049268124s) [noise=true , address=45.91.116.12:4226 , piece_id=MOFNZDLYQBXBBN7DDTF4T22K364CQDLS2DTICEPEGXYW4OXY47RA, seg_pos={0 0}, upload=471314]: Long tail cancellation due to optimal threshold Dec 28 00:53:20.178201261 (537h24m20.049284306s) [seg_pos={0 0}, upload=471314]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:53:20.178293467 (537h24m20.049376502s) [upload=471314]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:53:20.178299308 (537h24m20.049382343s) [upload=471314]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:53:20.466730187 (537h24m20.337813252s) [upload=471314]: Done uploading segments... Dec 28 00:53:20.466756076 (537h24m20.337839111s) [upload=471314]: Upload finished. err: <nil> Duration: 30.002670039s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [...X.XXX.X.X.XX.X.XXX...X.X.XX.XXX....X...X.X.X...XXXXX.X.XX] duration: 30.003316271s trace: https://jaeger.dev.storj.io/trace/708eb75639035a8c FAIL (2025-12-28 00:53:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 00:52:47.359336763 (537h23m47.230419798s) [upload=471306]: Starting upload Dec 28 00:52:47.359401146 (537h23m47.230484181s) [upload=471306]: Done starting upload Dec 28 00:52:47.359491108 (537h23m47.230574143s) [upload=471306]: Uploading segments... Dec 28 00:52:47.359581650 (537h23m47.230664685s) [upload=471306]: Got next segment. Inline: false Dec 28 00:52:47.359608050 (537h23m47.230691085s) [upload=471306]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:52:47.359613971 (537h23m47.230697006s) [upload=471306]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:52:47.600479008 (537h23m47.471562043s) [seg_pos={0 0}, upload=471306]: Begin upload segment... Dec 28 00:52:47.600532019 (537h23m47.471615064s) [seg_pos={0 0}, upload=471306]: Done begin upload segment. Dec 28 00:52:47.600563359 (537h23m47.471646404s) [noise=true , address=145.79.227.47:4267 , piece_id=YXH5RFKI3GNLZ3YIYKGGQVYVEL2GWOAVU3T2W6WOKMLITC3KKMUQ, seg_pos={0 0}, upload=471306]: Uploading piece... Dec 28 00:52:47.600593847 (537h23m47.471676882s) [noise=true , address=77.90.31.21:4878 , piece_id=PTNDJHQAIP37KRSOPXEL55RYSET5CTKK4FPOEVUIDFYEKNL5DK2Q, seg_pos={0 0}, upload=471306]: Uploading piece... Dec 28 00:52:47.601081305 (537h23m47.47216434s) [noise=true , address=145.79.227.42:4123 , piece_id=BOW7P7FSCFRIZZUQ6Z2DHRNV7FQPB5KUHIFFPA6Z2G7CPAEVXFAQ, seg_pos={0 0}, upload=471306]: Uploading piece... Dec 28 00:52:47.601474413 (537h23m47.472557459s) [noise=true , address=45.91.116.12:4225 , piece_id=PEUJCVV2DODGEGXN6WFVSOFXH3UYHQPSKRT5OIGK5ZN26EPIUZKQ, seg_pos={0 0}, upload=471306]: Uploading piece... Dec 28 00:52:47.601937836 (537h23m47.473020871s) [noise=true , address=145.79.227.18:4007 , piece_id=O7DNTMFXJIJPBQAZQNY3L7GAJYYVBZ4LTFVP5E5JMUHHYIP2SCTQ, seg_pos={0 0}, upload=471306]: Uploading piece... Dec 28 00:52:47.602328059 (537h23m47.473476097s) [noise=true , address=145.79.227.5:4223 , piece_id=7PANZFYGH2TMSH7HMNUXZXXD775PHFV4LXZ3XG2BNICMBTRJ3DNA, seg_pos={0 0}, upload=471306]: Uploading piece... Dec 28 00:52:47.602793585 (537h23m47.473876631s) [noise=true , address=45.91.116.6:4973 , piece_id=63C2VPR37BWBYA2DFM7LGQ4UPEQCNDHPMJ6RHOJ2S4U4AKH4K4YQ, seg_pos={0 0}, upload=471306]: Uploading piece... Dec 28 00:52:47.603194969 (537h23m47.474278004s) [noise=true , address=145.79.227.52:4446 , piece_id=G4CYPJKTJJCV6P7KP5B3PWWLILQDNUMAR6XERJT5QX4BD7AG3NIQ, seg_pos={0 0}, upload=471306]: Uploading piece... Dec 28 00:52:47.603626421 (537h23m47.474709456s) [noise=true , address=45.91.116.13:4156 , piece_id=RZQLRWJYVUUN74KILOAIBZCAA5LJNPU44ZELG5CYHN73GHGVBG2A, seg_pos={0 0}, upload=471306]: Uploading piece... Dec 28 00:52:47.605542981 (537h23m47.476626026s) [noise=true , address=145.79.227.37:4567 , piece_id=JNN2IMAUY3UILQLOMX5PLMY7XU4I4CJCZDLB5ZDAFPWOZI2B4PSA, seg_pos={0 0}, upload=471306]: Uploading piece... Dec 28 00:52:47.607444282 (537h23m47.478527317s) [noise=true , address=45.91.116.11:4117 , piece_id=M6IB6PGYXCFNCF26R3HYSBAJYMRPA6IAZU73WAGOH2ON2Q5AJEVQ, seg_pos={0 0}, upload=471306]: Uploading piece... Dec 28 00:52:47.609215523 (537h23m47.480298569s) [noise=true , address=145.79.227.17:4970 , piece_id=SEIVKYWAZVVNANA4LQAESMRUGIR4LXR3EFMNKNMDZY7Y2KPIQHIA, seg_pos={0 0}, upload=471306]: Uploading piece... Dec 28 00:52:47.610837733 (537h23m47.481920778s) [noise=true , address=45.91.116.10:4084 , piece_id=HEHX7TL2ZBOBXDPUFRMVZXSEDJPMWN2ESJVHOGB42WICX6NOZ33Q, seg_pos={0 0}, upload=471306]: Uploading piece... Dec 28 00:52:47.612350693 (537h23m47.483433738s) [noise=true , address=145.79.227.15:4875 , piece_id=7GBN3KRYHOQXYRBFRY7U4CFRCQO7Y6A4ILJOBNQDPFXSZEEKK27Q, seg_pos={0 0}, upload=471306]: Uploading piece... Dec 28 00:52:47.612963571 (537h23m47.484046616s) [noise=true , address=45.91.116.14:4190 , piece_id=M7RMUNKLAYGIBCG72QKJMEYDDEW6TCQRBK2P3IU5YQLWZGCAHOHQ, seg_pos={0 0}, upload=471306]: Uploading piece... Dec 28 00:52:47.614289896 (537h23m47.485372931s) [noise=true , address=145.79.227.31:4798 , piece_id=AOSC4E6A6FSEKVWDMX7QH4MHJR7DGIDQBOYYF2R4PFZ5B5GFKDPQ, seg_pos={0 0}, upload=471306]: Uploading piece... Dec 28 00:52:49.312692584 (537h23m49.183775619s) [upload=471306]: Next returned nil segment Dec 28 00:52:49.312710978 (537h23m49.183794013s) [upload=471306]: Waiting for error group managing segments... Dec 28 00:52:49.593196308 (537h23m49.464279363s) [noise=true , address=145.79.227.52:4446 , piece_id=G4CYPJKTJJCV6P7KP5B3PWWLILQDNUMAR6XERJT5QX4BD7AG3NIQ, seg_pos={0 0}, upload=471306]: Done uploading piece. err: <nil> Dec 28 00:52:49.607920475 (537h23m49.479003511s) [noise=true , address=145.79.227.5:4223 , piece_id=7PANZFYGH2TMSH7HMNUXZXXD775PHFV4LXZ3XG2BNICMBTRJ3DNA, seg_pos={0 0}, upload=471306]: Done uploading piece. err: <nil> Dec 28 00:52:49.608503756 (537h23m49.479586791s) [noise=true , address=145.79.227.18:4007 , piece_id=O7DNTMFXJIJPBQAZQNY3L7GAJYYVBZ4LTFVP5E5JMUHHYIP2SCTQ, seg_pos={0 0}, upload=471306]: Done uploading piece. err: <nil> Dec 28 00:52:49.611487658 (537h23m49.482570693s) [noise=true , address=145.79.227.37:4567 , piece_id=JNN2IMAUY3UILQLOMX5PLMY7XU4I4CJCZDLB5ZDAFPWOZI2B4PSA, seg_pos={0 0}, upload=471306]: Done uploading piece. err: <nil> Dec 28 00:52:49.615678298 (537h23m49.486761343s) [noise=true , address=145.79.227.31:4798 , piece_id=AOSC4E6A6FSEKVWDMX7QH4MHJR7DGIDQBOYYF2R4PFZ5B5GFKDPQ, seg_pos={0 0}, upload=471306]: Done uploading piece. err: <nil> Dec 28 00:52:49.617878427 (537h23m49.488961462s) [noise=true , address=145.79.227.17:4970 , piece_id=SEIVKYWAZVVNANA4LQAESMRUGIR4LXR3EFMNKNMDZY7Y2KPIQHIA, seg_pos={0 0}, upload=471306]: Done uploading piece. err: <nil> Dec 28 00:52:49.625630799 (537h23m49.496713834s) [noise=true , address=145.79.227.42:4123 , piece_id=BOW7P7FSCFRIZZUQ6Z2DHRNV7FQPB5KUHIFFPA6Z2G7CPAEVXFAQ, seg_pos={0 0}, upload=471306]: Done uploading piece. err: <nil> Dec 28 00:52:49.632602184 (537h23m49.503685219s) [noise=true , address=145.79.227.15:4875 , piece_id=7GBN3KRYHOQXYRBFRY7U4CFRCQO7Y6A4ILJOBNQDPFXSZEEKK27Q, seg_pos={0 0}, upload=471306]: Done uploading piece. err: <nil> Dec 28 00:52:49.645619371 (537h23m49.516702406s) [noise=true , address=145.79.227.47:4267 , piece_id=YXH5RFKI3GNLZ3YIYKGGQVYVEL2GWOAVU3T2W6WOKMLITC3KKMUQ, seg_pos={0 0}, upload=471306]: Done uploading piece. err: <nil> Dec 28 00:52:49.821062483 (537h23m49.692145518s) [noise=true , address=45.91.116.14:4190 , piece_id=M7RMUNKLAYGIBCG72QKJMEYDDEW6TCQRBK2P3IU5YQLWZGCAHOHQ, seg_pos={0 0}, upload=471306]: Done uploading piece. err: <nil> Dec 28 00:52:49.822478129 (537h23m49.693561164s) [noise=true , address=45.91.116.6:4973 , piece_id=63C2VPR37BWBYA2DFM7LGQ4UPEQCNDHPMJ6RHOJ2S4U4AKH4K4YQ, seg_pos={0 0}, upload=471306]: Done uploading piece. err: <nil> Dec 28 00:52:49.889949516 (537h23m49.761032561s) [noise=true , address=45.91.116.13:4156 , piece_id=RZQLRWJYVUUN74KILOAIBZCAA5LJNPU44ZELG5CYHN73GHGVBG2A, seg_pos={0 0}, upload=471306]: Done uploading piece. err: <nil> Dec 28 00:52:49.960834711 (537h23m49.831917746s) [noise=true , address=45.91.116.11:4117 , piece_id=M6IB6PGYXCFNCF26R3HYSBAJYMRPA6IAZU73WAGOH2ON2Q5AJEVQ, seg_pos={0 0}, upload=471306]: Done uploading piece. err: <nil> Dec 28 00:52:49.960854940 (537h23m49.831937975s) [seg_pos={0 0}, upload=471306]: Segment reached optimal threshold of 13 pieces. Dec 28 00:52:49.961035984 (537h23m49.83211905s) [noise=true , address=45.91.116.12:4225 , piece_id=PEUJCVV2DODGEGXN6WFVSOFXH3UYHQPSKRT5OIGK5ZN26EPIUZKQ, seg_pos={0 0}, upload=471306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:51668->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:51668->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51668->45.91.116.12:4225: use of closed network connection Dec 28 00:52:49.961046635 (537h23m49.832129671s) [noise=true , address=45.91.116.12:4225 , piece_id=PEUJCVV2DODGEGXN6WFVSOFXH3UYHQPSKRT5OIGK5ZN26EPIUZKQ, seg_pos={0 0}, upload=471306]: Long tail cancellation due to optimal threshold Dec 28 00:52:49.961162756 (537h23m49.832245801s) [noise=true , address=45.91.116.10:4084 , piece_id=HEHX7TL2ZBOBXDPUFRMVZXSEDJPMWN2ESJVHOGB42WICX6NOZ33Q, seg_pos={0 0}, upload=471306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 28 00:52:49.961188536 (537h23m49.832271571s) [noise=true , address=45.91.116.10:4084 , piece_id=HEHX7TL2ZBOBXDPUFRMVZXSEDJPMWN2ESJVHOGB42WICX6NOZ33Q, seg_pos={0 0}, upload=471306]: Long tail cancellation due to optimal threshold Dec 28 00:52:49.961316829 (537h23m49.832399864s) [noise=true , address=77.90.31.21:4878 , piece_id=PTNDJHQAIP37KRSOPXEL55RYSET5CTKK4FPOEVUIDFYEKNL5DK2Q, seg_pos={0 0}, upload=471306]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 00:52:49.961323091 (537h23m49.832406126s) [noise=true , address=77.90.31.21:4878 , piece_id=PTNDJHQAIP37KRSOPXEL55RYSET5CTKK4FPOEVUIDFYEKNL5DK2Q, seg_pos={0 0}, upload=471306]: Long tail cancellation due to optimal threshold Dec 28 00:52:49.961345704 (537h23m49.832428739s) [seg_pos={0 0}, upload=471306]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:52:49.961404577 (537h23m49.832487612s) [upload=471306]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:52:49.961408945 (537h23m49.83249198s) [upload=471306]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:52:50.252714419 (537h23m50.123797454s) [upload=471306]: Done uploading segments... Dec 28 00:52:50.252733645 (537h23m50.12381668s) [upload=471306]: Upload finished. err: <nil> Duration: 30.003326722s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 26.923912933s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 00:52:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 26.923932209s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [....X.XXX.X.X.XX.X.XXX...X.X.XX.XXX....X...X.X.X...XXXXX.X.X] duration: 30.003346207s trace: https://jaeger.dev.storj.io/trace/47c5c3723baca7db FAIL (2025-12-28 00:52:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 00:52:17.135722231 (537h23m17.006805266s) [upload=471301]: Starting upload Dec 28 00:52:17.135803355 (537h23m17.00688639s) [upload=471301]: Done starting upload Dec 28 00:52:17.135879921 (537h23m17.006962976s) [upload=471301]: Uploading segments... Dec 28 00:52:17.136034265 (537h23m17.00711731s) [upload=471301]: Got next segment. Inline: false Dec 28 00:52:17.136068831 (537h23m17.007151866s) [upload=471301]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:52:17.136074051 (537h23m17.007157086s) [upload=471301]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:52:17.376948455 (537h23m17.24803149s) [seg_pos={0 0}, upload=471301]: Begin upload segment... Dec 28 00:52:17.377039659 (537h23m17.248122694s) [seg_pos={0 0}, upload=471301]: Done begin upload segment. Dec 28 00:52:17.377062713 (537h23m17.248145748s) [noise=true , address=145.79.227.9:4367 , piece_id=MWYOCHRIWT5TFT3NAGMXBRQXPEBNEFHNVRNKLCE6ZPWFRFOTWKQA, seg_pos={0 0}, upload=471301]: Uploading piece... Dec 28 00:52:17.377101116 (537h23m17.248184151s) [noise=true , address=145.79.227.45:4230 , piece_id=YR3TTTRONDH6J2LL3QC6KCON63DYH6C6AW7F74BD5UQWIQLULEUQ, seg_pos={0 0}, upload=471301]: Uploading piece... Dec 28 00:52:17.377836747 (537h23m17.248919792s) [noise=true , address=145.79.227.34:4906 , piece_id=XHDUXXMVO7PJO6SZXGMAAKYPDKMBPZW3APLLWH4FHR4IA5RFUGCQ, seg_pos={0 0}, upload=471301]: Uploading piece... Dec 28 00:52:17.378303385 (537h23m17.24938642s) [noise=true , address=77.90.31.15:4700 , piece_id=6EU7II5ZQPYHIJNXGX6CSBWPPNJ6KQGZKGDF3IDLYR7ELNVIZ2RQ, seg_pos={0 0}, upload=471301]: Uploading piece... Dec 28 00:52:17.378906233 (537h23m17.249989268s) [noise=true , address=145.79.227.19:4077 , piece_id=XCB5JZA23RMKACPWJ25NPHSXQVUYZGSDN46XU4OJ5ROLWER45AIQ, seg_pos={0 0}, upload=471301]: Uploading piece... Dec 28 00:52:17.379374665 (537h23m17.25045771s) [noise=true , address=145.79.227.37:4566 , piece_id=JDPV77NA47A7HT6NESM3NUACXZGVQ45U6ODFPH6IUSKAFRXXWREA, seg_pos={0 0}, upload=471301]: Uploading piece... Dec 28 00:52:17.379877432 (537h23m17.250960467s) [noise=true , address=45.91.116.8:4045 , piece_id=RYAV5W76SEZF6ZMJDJ34SK7NV3LZK6FJB2DLX4IH6ZLTWV6ZCFOA, seg_pos={0 0}, upload=471301]: Uploading piece... Dec 28 00:52:17.380321648 (537h23m17.251404683s) [noise=true , address=145.79.227.31:4798 , piece_id=YW6EI7AYQG54OTO3B5FPQOEVEYYCFOJNRNILYJKWX3HB5QM7AR3Q, seg_pos={0 0}, upload=471301]: Uploading piece... Dec 28 00:52:17.380804648 (537h23m17.251887693s) [noise=true , address=45.91.116.14:4191 , piece_id=P7C3WUPLCNSCFAQAIXWW6J5OGRLG7MIEPHNHYJVWZ5B6GQXRFMDA, seg_pos={0 0}, upload=471301]: Uploading piece... Dec 28 00:52:17.381160696 (537h23m17.252243731s) [noise=true , address=45.91.116.6:4973 , piece_id=GKRQJZK4BN22TQZ5N4B7L7YUIB5LE2UQ5HNCNNYGRFAZMDOLNWFA, seg_pos={0 0}, upload=471301]: Uploading piece... Dec 28 00:52:17.381471317 (537h23m17.252554352s) [noise=true , address=145.79.227.12:4765 , piece_id=R7P4O3J2VVLONXTA7GWOYDPG6HOPYF2Z5XUWFHZSBET4JJEKS7DA, seg_pos={0 0}, upload=471301]: Uploading piece... Dec 28 00:52:17.382733150 (537h23m17.253816195s) [noise=true , address=145.79.227.16:4911 , piece_id=OSOUQ35KVDTYRSZNI5TDHTHWMJMGNDZLNONAKVXCFXC4URBKWB2Q, seg_pos={0 0}, upload=471301]: Uploading piece... Dec 28 00:52:17.383014376 (537h23m17.254097421s) [noise=true , address=145.79.227.47:4266 , piece_id=IN2L2F4GYUARBLUKUYVTMGRR2KA3IXB5RLH6OCPFYLUXGFQ2UR2A, seg_pos={0 0}, upload=471301]: Uploading piece... Dec 28 00:52:17.384138717 (537h23m17.255221752s) [noise=true , address=145.79.227.52:4445 , piece_id=PXGUP24MMQNOQOOG2CNPBCBVLFOQD6PI4MX6S7VJDWCQNDDRFHJQ, seg_pos={0 0}, upload=471301]: Uploading piece... Dec 28 00:52:17.385323422 (537h23m17.256406468s) [noise=true , address=145.79.227.36:4518 , piece_id=MWCXCRSAY6B3Z3ZENFP6RHIUR3IVZGJ5ACDJA5ONDPKH4INKMWTA, seg_pos={0 0}, upload=471301]: Uploading piece... Dec 28 00:52:17.386702920 (537h23m17.257785955s) [noise=true , address=45.91.116.12:4228 , piece_id=JO3QXHYWHTQ6UO4YXGBGQ5WKUMWVC3FX2SIDC4IZ76F4WOXEPNQA, seg_pos={0 0}, upload=471301]: Uploading piece... Dec 28 00:52:19.099667165 (537h23m18.97075021s) [upload=471301]: Next returned nil segment Dec 28 00:52:19.099691070 (537h23m18.970774115s) [upload=471301]: Waiting for error group managing segments... Dec 28 00:52:19.386244696 (537h23m19.257327741s) [noise=true , address=145.79.227.12:4765 , piece_id=R7P4O3J2VVLONXTA7GWOYDPG6HOPYF2Z5XUWFHZSBET4JJEKS7DA, seg_pos={0 0}, upload=471301]: Done uploading piece. err: <nil> Dec 28 00:52:19.387712541 (537h23m19.258795576s) [noise=true , address=145.79.227.36:4518 , piece_id=MWCXCRSAY6B3Z3ZENFP6RHIUR3IVZGJ5ACDJA5ONDPKH4INKMWTA, seg_pos={0 0}, upload=471301]: Done uploading piece. err: <nil> Dec 28 00:52:19.388003735 (537h23m19.25908677s) [noise=true , address=145.79.227.45:4230 , piece_id=YR3TTTRONDH6J2LL3QC6KCON63DYH6C6AW7F74BD5UQWIQLULEUQ, seg_pos={0 0}, upload=471301]: Done uploading piece. err: <nil> Dec 28 00:52:19.393358752 (537h23m19.264441807s) [noise=true , address=145.79.227.31:4798 , piece_id=YW6EI7AYQG54OTO3B5FPQOEVEYYCFOJNRNILYJKWX3HB5QM7AR3Q, seg_pos={0 0}, upload=471301]: Done uploading piece. err: <nil> Dec 28 00:52:19.405272007 (537h23m19.276355052s) [noise=true , address=145.79.227.9:4367 , piece_id=MWYOCHRIWT5TFT3NAGMXBRQXPEBNEFHNVRNKLCE6ZPWFRFOTWKQA, seg_pos={0 0}, upload=471301]: Done uploading piece. err: <nil> Dec 28 00:52:19.405342932 (537h23m19.276425977s) [noise=true , address=145.79.227.52:4445 , piece_id=PXGUP24MMQNOQOOG2CNPBCBVLFOQD6PI4MX6S7VJDWCQNDDRFHJQ, seg_pos={0 0}, upload=471301]: Done uploading piece. err: <nil> Dec 28 00:52:19.412559583 (537h23m19.283642618s) [noise=true , address=145.79.227.37:4566 , piece_id=JDPV77NA47A7HT6NESM3NUACXZGVQ45U6ODFPH6IUSKAFRXXWREA, seg_pos={0 0}, upload=471301]: Done uploading piece. err: <nil> Dec 28 00:52:19.421288234 (537h23m19.292371269s) [noise=true , address=145.79.227.19:4077 , piece_id=XCB5JZA23RMKACPWJ25NPHSXQVUYZGSDN46XU4OJ5ROLWER45AIQ, seg_pos={0 0}, upload=471301]: Done uploading piece. err: <nil> Dec 28 00:52:19.421672977 (537h23m19.292756012s) [noise=true , address=145.79.227.16:4911 , piece_id=OSOUQ35KVDTYRSZNI5TDHTHWMJMGNDZLNONAKVXCFXC4URBKWB2Q, seg_pos={0 0}, upload=471301]: Done uploading piece. err: <nil> Dec 28 00:52:19.426927062 (537h23m19.298010097s) [noise=true , address=145.79.227.47:4266 , piece_id=IN2L2F4GYUARBLUKUYVTMGRR2KA3IXB5RLH6OCPFYLUXGFQ2UR2A, seg_pos={0 0}, upload=471301]: Done uploading piece. err: <nil> Dec 28 00:52:19.620948981 (537h23m19.492032016s) [noise=true , address=45.91.116.14:4191 , piece_id=P7C3WUPLCNSCFAQAIXWW6J5OGRLG7MIEPHNHYJVWZ5B6GQXRFMDA, seg_pos={0 0}, upload=471301]: Done uploading piece. err: <nil> Dec 28 00:52:19.621400401 (537h23m19.492483436s) [noise=true , address=77.90.31.15:4700 , piece_id=6EU7II5ZQPYHIJNXGX6CSBWPPNJ6KQGZKGDF3IDLYR7ELNVIZ2RQ, seg_pos={0 0}, upload=471301]: Done uploading piece. err: <nil> Dec 28 00:52:19.622225602 (537h23m19.493308648s) [noise=true , address=45.91.116.6:4973 , piece_id=GKRQJZK4BN22TQZ5N4B7L7YUIB5LE2UQ5HNCNNYGRFAZMDOLNWFA, seg_pos={0 0}, upload=471301]: Done uploading piece. err: <nil> Dec 28 00:52:19.622233708 (537h23m19.493316743s) [seg_pos={0 0}, upload=471301]: Segment reached optimal threshold of 13 pieces. Dec 28 00:52:19.622474717 (537h23m19.493557752s) [noise=true , address=45.91.116.12:4228 , piece_id=JO3QXHYWHTQ6UO4YXGBGQ5WKUMWVC3FX2SIDC4IZ76F4WOXEPNQA, seg_pos={0 0}, upload=471301]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 28 00:52:19.622481660 (537h23m19.493564705s) [noise=true , address=45.91.116.12:4228 , piece_id=JO3QXHYWHTQ6UO4YXGBGQ5WKUMWVC3FX2SIDC4IZ76F4WOXEPNQA, seg_pos={0 0}, upload=471301]: Long tail cancellation due to optimal threshold Dec 28 00:52:19.622624161 (537h23m19.493707206s) [noise=true , address=145.79.227.34:4906 , piece_id=XHDUXXMVO7PJO6SZXGMAAKYPDKMBPZW3APLLWH4FHR4IA5RFUGCQ, seg_pos={0 0}, upload=471301]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 28 00:52:19.622644370 (537h23m19.493727405s) [noise=true , address=145.79.227.34:4906 , piece_id=XHDUXXMVO7PJO6SZXGMAAKYPDKMBPZW3APLLWH4FHR4IA5RFUGCQ, seg_pos={0 0}, upload=471301]: Long tail cancellation due to optimal threshold Dec 28 00:52:19.622694214 (537h23m19.493777249s) [noise=true , address=45.91.116.8:4045 , piece_id=RYAV5W76SEZF6ZMJDJ34SK7NV3LZK6FJB2DLX4IH6ZLTWV6ZCFOA, seg_pos={0 0}, upload=471301]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 28 00:52:19.622702169 (537h23m19.493785204s) [noise=true , address=45.91.116.8:4045 , piece_id=RYAV5W76SEZF6ZMJDJ34SK7NV3LZK6FJB2DLX4IH6ZLTWV6ZCFOA, seg_pos={0 0}, upload=471301]: Long tail cancellation due to optimal threshold Dec 28 00:52:19.622742918 (537h23m19.493825953s) [seg_pos={0 0}, upload=471301]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:52:19.622876130 (537h23m19.493959176s) [upload=471301]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:52:19.622881582 (537h23m19.493964627s) [upload=471301]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:52:19.907721984 (537h23m19.77880502s) [upload=471301]: Done uploading segments... Dec 28 00:52:19.907745829 (537h23m19.778828884s) [upload=471301]: Upload finished. err: <nil> Duration: 30.003367727s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 34.888162245s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 00:52:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 34.888193265s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 27.600242765s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 00:51:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 27.600260429s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [X....X.XXX.X.X.XX.X.XXX...X.X.XX.XXX....X...X.X.X...XXXXX.X.] duration: 6.014521717s trace: https://jaeger.dev.storj.io/trace/600745e6cce550d3 FAIL (2025-12-28 00:51:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: KYROHTNXD6OZTPYWQFZMWNDJW4WHXCKT6LS4DSFYDEQHPXXNU45A) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: DU3GOV2655OIVVN2KM4CKK2NJSN2UR3KV5SKVOMAOV4UBXU5JTZQ) context deadline exceeded ======LOGS====== :39.369437441 (537h21m39.240520476s) [upload=471280]: Starting upload Dec 28 00:50:39.369509828 (537h21m39.240592873s) [upload=471280]: Done starting upload Dec 28 00:50:39.369697156 (537h21m39.240780191s) [upload=471280]: Uploading segments... Dec 28 00:50:39.369790252 (537h21m39.240873287s) [upload=471280]: Got next segment. Inline: false Dec 28 00:50:39.369873331 (537h21m39.240956376s) [upload=471280]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:50:39.369880004 (537h21m39.240963049s) [upload=471280]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:50:39.615102720 (537h21m39.486185765s) [seg_pos={0 0}, upload=471280]: Begin upload segment... Dec 28 00:50:39.615164688 (537h21m39.486247723s) [seg_pos={0 0}, upload=471280]: Done begin upload segment. Dec 28 00:50:39.615208631 (537h21m39.486291676s) [noise=true , address=77.90.31.13:4629 , piece_id=KYROHTNXD6OZTPYWQFZMWNDJW4WHXCKT6LS4DSFYDEQHPXXNU45A, seg_pos={0 0}, upload=471280]: Uploading piece... Dec 28 00:50:39.615303151 (537h21m39.486386186s) [noise=true , address=145.79.227.48:4303 , piece_id=VDY6RQ6QE7QVWE7NRQXO4Y52QY7H7O477FYYY3WG5AIG5O2H5T2Q, seg_pos={0 0}, upload=471280]: Uploading piece... Dec 28 00:50:39.615797172 (537h21m39.486880237s) [noise=true , address=77.90.31.14:4664 , piece_id=H4TGPIOVEFLW46ETMDAKCVUSGXQGB4MSKAOYC322IEKZOGLABOOQ, seg_pos={0 0}, upload=471280]: Uploading piece... Dec 28 00:50:39.616379822 (537h21m39.487462867s) [noise=true , address=45.91.116.12:4228 , piece_id=GKF7XDSC6KYCCXRP6IG7KRSDFS76NGFJTSDWG3MTDCLQRQDZZRJA, seg_pos={0 0}, upload=471280]: Uploading piece... Dec 28 00:50:39.616842392 (537h21m39.487925427s) [noise=true , address=145.79.227.41:4086 , piece_id=TF5Q6ZQ2WOL4MT3R4DDQDKY7N6D2E2OGUCGIYHWAKAKZ3TM363LA, seg_pos={0 0}, upload=471280]: Uploading piece... Dec 28 00:50:39.617217737 (537h21m39.488300772s) [noise=true , address=145.79.227.5:4223 , piece_id=ANA5H33UECXA6OWMHCH76IWFC3RYP562UVRWR47XMZ6U5P244U3Q, seg_pos={0 0}, upload=471280]: Uploading piece... Dec 28 00:50:39.617550059 (537h21m39.488633094s) [noise=true , address=45.91.116.7:4009 , piece_id=DU3GOV2655OIVVN2KM4CKK2NJSN2UR3KV5SKVOMAOV4UBXU5JTZQ, seg_pos={0 0}, upload=471280]: Uploading piece... Dec 28 00:50:39.618011498 (537h21m39.489094533s) [noise=true , address=145.79.227.40:4049 , piece_id=BX6R3HBU6CJBLO2O4TGV3OFWJTCPG6EOFLGH2FFN3YG3HALZV23A, seg_pos={0 0}, upload=471280]: Uploading piece... Dec 28 00:50:39.618702243 (537h21m39.489785278s) [noise=true , address=45.91.116.8:4045 , piece_id=BAXTL7SAIBHCFYT2XBFQT5RCHRTDYDC72NMNCXILX7CTY3TZZROA, seg_pos={0 0}, upload=471280]: Uploading piece... Dec 28 00:50:39.619426102 (537h21m39.490509147s) [noise=true , address=45.91.116.6:4974 , piece_id=BCPQNVYSJ5424XKFOAVBBBB3MCOQKLXLRNNGWX4ZVZGIAZPPLL7A, seg_pos={0 0}, upload=471280]: Uploading piece... Dec 28 00:50:39.620298142 (537h21m39.491381207s) [noise=true , address=45.91.116.4:4940 , piece_id=KF245Y4QODMKKKHGFRODVCYT5WIK3JA7DNZBDIYWPC6REKPASIQA, seg_pos={0 0}, upload=471280]: Uploading piece... Dec 28 00:50:39.620848460 (537h21m39.491931495s) [noise=true , address=145.79.227.15:4876 , piece_id=GWWTUV5EJBZUAQGYCL4AATZPY6AWND66NQBVDBN2KSZE5EGLWF4Q, seg_pos={0 0}, upload=471280]: Uploading piece... Dec 28 00:50:39.621760817 (537h21m39.492843852s) [noise=true , address=145.79.227.11:4730 , piece_id=IAIV24YA4NT5DXNZNS6LYBCL3KPLKVN4YGR3D2VVKDWSDRAKP3ZQ, seg_pos={0 0}, upload=471280]: Uploading piece... Dec 28 00:50:39.632828663 (537h21m39.503911698s) [noise=true , address=45.91.116.5:4297 , piece_id=H5WZYQNHZN2GW3AZISJCYV24RMIWVOMG6PMIJVFKNSKKCFID3P3A, seg_pos={0 0}, upload=471280]: Uploading piece... Dec 28 00:50:39.634065258 (537h21m39.505148293s) [noise=true , address=145.79.227.24:4583 , piece_id=EGXHECV6ZYXAU3POC4JXKZ4OBSQUB5JYGG6DIL2VQYG52O7RGMYQ, seg_pos={0 0}, upload=471280]: Uploading piece... Dec 28 00:50:39.635892146 (537h21m39.506975191s) [noise=true , address=45.91.116.11:4119 , piece_id=QRFSTIPKYGVJ3EEGCK2R4TUWAYGPUZYMUMIFX3446LDECX4MO2KA, seg_pos={0 0}, upload=471280]: Uploading piece... Dec 28 00:50:41.346592953 (537h21m41.217675998s) [upload=471280]: Next returned nil segment Dec 28 00:50:41.346606919 (537h21m41.217689964s) [upload=471280]: Waiting for error group managing segments... Dec 28 00:50:41.620419753 (537h21m41.491502788s) [noise=true , address=145.79.227.5:4223 , piece_id=ANA5H33UECXA6OWMHCH76IWFC3RYP562UVRWR47XMZ6U5P244U3Q, seg_pos={0 0}, upload=471280]: Done uploading piece. err: <nil> Dec 28 00:50:41.629055677 (537h21m41.500138712s) [noise=true , address=145.79.227.11:4730 , piece_id=IAIV24YA4NT5DXNZNS6LYBCL3KPLKVN4YGR3D2VVKDWSDRAKP3ZQ, seg_pos={0 0}, upload=471280]: Done uploading piece. err: <nil> Dec 28 00:50:41.629459517 (537h21m41.500542562s) [noise=true , address=145.79.227.15:4876 , piece_id=GWWTUV5EJBZUAQGYCL4AATZPY6AWND66NQBVDBN2KSZE5EGLWF4Q, seg_pos={0 0}, upload=471280]: Done uploading piece. err: <nil> Dec 28 00:50:41.659717032 (537h21m41.530800067s) [noise=true , address=145.79.227.24:4583 , piece_id=EGXHECV6ZYXAU3POC4JXKZ4OBSQUB5JYGG6DIL2VQYG52O7RGMYQ, seg_pos={0 0}, upload=471280]: Done uploading piece. err: <nil> Dec 28 00:50:41.871557421 (537h21m41.742640466s) [noise=true , address=45.91.116.5:4297 , piece_id=H5WZYQNHZN2GW3AZISJCYV24RMIWVOMG6PMIJVFKNSKKCFID3P3A, seg_pos={0 0}, upload=471280]: Done uploading piece. err: <nil> Dec 28 00:50:41.877523411 (537h21m41.748606446s) [noise=true , address=77.90.31.13:4629 , piece_id=KYROHTNXD6OZTPYWQFZMWNDJW4WHXCKT6LS4DSFYDEQHPXXNU45A, seg_pos={0 0}, upload=471280]: Done uploading piece. err: <nil> Dec 28 00:50:41.975310333 (537h21m41.846393378s) [noise=true , address=45.91.116.11:4119 , piece_id=QRFSTIPKYGVJ3EEGCK2R4TUWAYGPUZYMUMIFX3446LDECX4MO2KA, seg_pos={0 0}, upload=471280]: Done uploading piece. err: <nil> Dec 28 00:50:41.981593527 (537h21m41.852676562s) [noise=true , address=45.91.116.4:4940 , piece_id=KF245Y4QODMKKKHGFRODVCYT5WIK3JA7DNZBDIYWPC6REKPASIQA, seg_pos={0 0}, upload=471280]: Done uploading piece. err: <nil> Dec 28 00:50:41.990893425 (537h21m41.86197646s) [noise=true , address=45.91.116.6:4974 , piece_id=BCPQNVYSJ5424XKFOAVBBBB3MCOQKLXLRNNGWX4ZVZGIAZPPLL7A, seg_pos={0 0}, upload=471280]: Done uploading piece. err: <nil> Dec 28 00:50:42.001694684 (537h21m41.872777739s) [noise=true , address=145.79.227.41:4086 , piece_id=TF5Q6ZQ2WOL4MT3R4DDQDKY7N6D2E2OGUCGIYHWAKAKZ3TM363LA, seg_pos={0 0}, upload=471280]: Done uploading piece. err: <nil> Dec 28 00:50:42.201683575 (537h21m42.07276661s) [noise=true , address=45.91.116.7:4009 , piece_id=DU3GOV2655OIVVN2KM4CKK2NJSN2UR3KV5SKVOMAOV4UBXU5JTZQ, seg_pos={0 0}, upload=471280]: Done uploading piece. err: <nil> Dec 28 00:50:42.631604345 (537h21m42.50268738s) [noise=true , address=45.91.116.8:4045 , piece_id=BAXTL7SAIBHCFYT2XBFQT5RCHRTDYDC72NMNCXILX7CTY3TZZROA, seg_pos={0 0}, upload=471280]: Done uploading piece. err: <nil> Dec 28 00:50:43.604727923 (537h21m43.475810968s) [noise=true , address=145.79.227.40:4049 , piece_id=BX6R3HBU6CJBLO2O4TGV3OFWJTCPG6EOFLGH2FFN3YG3HALZV23A, seg_pos={0 0}, upload=471280]: Done uploading piece. err: <nil> Dec 28 00:50:43.604741298 (537h21m43.475824343s) [seg_pos={0 0}, upload=471280]: Segment reached optimal threshold of 13 pieces. Dec 28 00:50:43.604969132 (537h21m43.476052167s) [noise=true , address=45.91.116.12:4228 , piece_id=GKF7XDSC6KYCCXRP6IG7KRSDFS76NGFJTSDWG3MTDCLQRQDZZRJA, seg_pos={0 0}, upload=471280]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 28 00:50:43.605009148 (537h21m43.476092183s) [noise=true , address=45.91.116.12:4228 , piece_id=GKF7XDSC6KYCCXRP6IG7KRSDFS76NGFJTSDWG3MTDCLQRQDZZRJA, seg_pos={0 0}, upload=471280]: Long tail cancellation due to optimal threshold Dec 28 00:50:43.605234877 (537h21m43.476317913s) [noise=true , address=77.90.31.14:4664 , piece_id=H4TGPIOVEFLW46ETMDAKCVUSGXQGB4MSKAOYC322IEKZOGLABOOQ, seg_pos={0 0}, upload=471280]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 28 00:50:43.605241200 (537h21m43.476324235s) [noise=true , address=77.90.31.14:4664 , piece_id=H4TGPIOVEFLW46ETMDAKCVUSGXQGB4MSKAOYC322IEKZOGLABOOQ, seg_pos={0 0}, upload=471280]: Long tail cancellation due to optimal threshold Dec 28 00:50:43.605395073 (537h21m43.476478108s) [noise=true , address=145.79.227.48:4303 , piece_id=VDY6RQ6QE7QVWE7NRQXO4Y52QY7H7O477FYYY3WG5AIG5O2H5T2Q, seg_pos={0 0}, upload=471280]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: write tcp 188.166.225.170:40142->145.79.227.48:4303: use of closed network connection; write tcp 188.166.225.170:40142->145.79.227.48:4303: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40142->145.79.227.48:4303: use of closed network connection Dec 28 00:50:43.605400814 (537h21m43.476483859s) [noise=true , address=145.79.227.48:4303 , piece_id=VDY6RQ6QE7QVWE7NRQXO4Y52QY7H7O477FYYY3WG5AIG5O2H5T2Q, seg_pos={0 0}, upload=471280]: Long tail cancellation due to optimal threshold Dec 28 00:50:43.605434969 (537h21m43.476518004s) [seg_pos={0 0}, upload=471280]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:50:43.605525602 (537h21m43.476608647s) [upload=471280]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:50:43.605530972 (537h21m43.476614007s) [upload=471280]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:50:43.902901053 (537h21m43.773984088s) [upload=471280]: Done uploading segments... Dec 28 00:50:43.902948082 (537h21m43.774031118s) [upload=471280]: Upload finished. err: <nil> Duration: 6.01453886s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [.X....X.XXX.X.X.XX.X.XXX...X.X.XX.XXX....X...X.X.X...XXXXX.X] duration: 30.00297084s trace: https://jaeger.dev.storj.io/trace/600745e6cce550d3 FAIL (2025-12-28 00:51:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: KYROHTNXD6OZTPYWQFZMWNDJW4WHXCKT6LS4DSFYDEQHPXXNU45A) context deadline exceeded; piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: DU3GOV2655OIVVN2KM4CKK2NJSN2UR3KV5SKVOMAOV4UBXU5JTZQ) context deadline exceeded ======LOGS====== :39.369437441 (537h21m39.240520476s) [upload=471280]: Starting upload Dec 28 00:50:39.369509828 (537h21m39.240592873s) [upload=471280]: Done starting upload Dec 28 00:50:39.369697156 (537h21m39.240780191s) [upload=471280]: Uploading segments... Dec 28 00:50:39.369790252 (537h21m39.240873287s) [upload=471280]: Got next segment. Inline: false Dec 28 00:50:39.369873331 (537h21m39.240956376s) [upload=471280]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:50:39.369880004 (537h21m39.240963049s) [upload=471280]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:50:39.615102720 (537h21m39.486185765s) [seg_pos={0 0}, upload=471280]: Begin upload segment... Dec 28 00:50:39.615164688 (537h21m39.486247723s) [seg_pos={0 0}, upload=471280]: Done begin upload segment. Dec 28 00:50:39.615208631 (537h21m39.486291676s) [noise=true , address=77.90.31.13:4629 , piece_id=KYROHTNXD6OZTPYWQFZMWNDJW4WHXCKT6LS4DSFYDEQHPXXNU45A, seg_pos={0 0}, upload=471280]: Uploading piece... Dec 28 00:50:39.615303151 (537h21m39.486386186s) [noise=true , address=145.79.227.48:4303 , piece_id=VDY6RQ6QE7QVWE7NRQXO4Y52QY7H7O477FYYY3WG5AIG5O2H5T2Q, seg_pos={0 0}, upload=471280]: Uploading piece... Dec 28 00:50:39.615797172 (537h21m39.486880237s) [noise=true , address=77.90.31.14:4664 , piece_id=H4TGPIOVEFLW46ETMDAKCVUSGXQGB4MSKAOYC322IEKZOGLABOOQ, seg_pos={0 0}, upload=471280]: Uploading piece... Dec 28 00:50:39.616379822 (537h21m39.487462867s) [noise=true , address=45.91.116.12:4228 , piece_id=GKF7XDSC6KYCCXRP6IG7KRSDFS76NGFJTSDWG3MTDCLQRQDZZRJA, seg_pos={0 0}, upload=471280]: Uploading piece... Dec 28 00:50:39.616842392 (537h21m39.487925427s) [noise=true , address=145.79.227.41:4086 , piece_id=TF5Q6ZQ2WOL4MT3R4DDQDKY7N6D2E2OGUCGIYHWAKAKZ3TM363LA, seg_pos={0 0}, upload=471280]: Uploading piece... Dec 28 00:50:39.617217737 (537h21m39.488300772s) [noise=true , address=145.79.227.5:4223 , piece_id=ANA5H33UECXA6OWMHCH76IWFC3RYP562UVRWR47XMZ6U5P244U3Q, seg_pos={0 0}, upload=471280]: Uploading piece... Dec 28 00:50:39.617550059 (537h21m39.488633094s) [noise=true , address=45.91.116.7:4009 , piece_id=DU3GOV2655OIVVN2KM4CKK2NJSN2UR3KV5SKVOMAOV4UBXU5JTZQ, seg_pos={0 0}, upload=471280]: Uploading piece... Dec 28 00:50:39.618011498 (537h21m39.489094533s) [noise=true , address=145.79.227.40:4049 , piece_id=BX6R3HBU6CJBLO2O4TGV3OFWJTCPG6EOFLGH2FFN3YG3HALZV23A, seg_pos={0 0}, upload=471280]: Uploading piece... Dec 28 00:50:39.618702243 (537h21m39.489785278s) [noise=true , address=45.91.116.8:4045 , piece_id=BAXTL7SAIBHCFYT2XBFQT5RCHRTDYDC72NMNCXILX7CTY3TZZROA, seg_pos={0 0}, upload=471280]: Uploading piece... Dec 28 00:50:39.619426102 (537h21m39.490509147s) [noise=true , address=45.91.116.6:4974 , piece_id=BCPQNVYSJ5424XKFOAVBBBB3MCOQKLXLRNNGWX4ZVZGIAZPPLL7A, seg_pos={0 0}, upload=471280]: Uploading piece... Dec 28 00:50:39.620298142 (537h21m39.491381207s) [noise=true , address=45.91.116.4:4940 , piece_id=KF245Y4QODMKKKHGFRODVCYT5WIK3JA7DNZBDIYWPC6REKPASIQA, seg_pos={0 0}, upload=471280]: Uploading piece... Dec 28 00:50:39.620848460 (537h21m39.491931495s) [noise=true , address=145.79.227.15:4876 , piece_id=GWWTUV5EJBZUAQGYCL4AATZPY6AWND66NQBVDBN2KSZE5EGLWF4Q, seg_pos={0 0}, upload=471280]: Uploading piece... Dec 28 00:50:39.621760817 (537h21m39.492843852s) [noise=true , address=145.79.227.11:4730 , piece_id=IAIV24YA4NT5DXNZNS6LYBCL3KPLKVN4YGR3D2VVKDWSDRAKP3ZQ, seg_pos={0 0}, upload=471280]: Uploading piece... Dec 28 00:50:39.632828663 (537h21m39.503911698s) [noise=true , address=45.91.116.5:4297 , piece_id=H5WZYQNHZN2GW3AZISJCYV24RMIWVOMG6PMIJVFKNSKKCFID3P3A, seg_pos={0 0}, upload=471280]: Uploading piece... Dec 28 00:50:39.634065258 (537h21m39.505148293s) [noise=true , address=145.79.227.24:4583 , piece_id=EGXHECV6ZYXAU3POC4JXKZ4OBSQUB5JYGG6DIL2VQYG52O7RGMYQ, seg_pos={0 0}, upload=471280]: Uploading piece... Dec 28 00:50:39.635892146 (537h21m39.506975191s) [noise=true , address=45.91.116.11:4119 , piece_id=QRFSTIPKYGVJ3EEGCK2R4TUWAYGPUZYMUMIFX3446LDECX4MO2KA, seg_pos={0 0}, upload=471280]: Uploading piece... Dec 28 00:50:41.346592953 (537h21m41.217675998s) [upload=471280]: Next returned nil segment Dec 28 00:50:41.346606919 (537h21m41.217689964s) [upload=471280]: Waiting for error group managing segments... Dec 28 00:50:41.620419753 (537h21m41.491502788s) [noise=true , address=145.79.227.5:4223 , piece_id=ANA5H33UECXA6OWMHCH76IWFC3RYP562UVRWR47XMZ6U5P244U3Q, seg_pos={0 0}, upload=471280]: Done uploading piece. err: <nil> Dec 28 00:50:41.629055677 (537h21m41.500138712s) [noise=true , address=145.79.227.11:4730 , piece_id=IAIV24YA4NT5DXNZNS6LYBCL3KPLKVN4YGR3D2VVKDWSDRAKP3ZQ, seg_pos={0 0}, upload=471280]: Done uploading piece. err: <nil> Dec 28 00:50:41.629459517 (537h21m41.500542562s) [noise=true , address=145.79.227.15:4876 , piece_id=GWWTUV5EJBZUAQGYCL4AATZPY6AWND66NQBVDBN2KSZE5EGLWF4Q, seg_pos={0 0}, upload=471280]: Done uploading piece. err: <nil> Dec 28 00:50:41.659717032 (537h21m41.530800067s) [noise=true , address=145.79.227.24:4583 , piece_id=EGXHECV6ZYXAU3POC4JXKZ4OBSQUB5JYGG6DIL2VQYG52O7RGMYQ, seg_pos={0 0}, upload=471280]: Done uploading piece. err: <nil> Dec 28 00:50:41.871557421 (537h21m41.742640466s) [noise=true , address=45.91.116.5:4297 , piece_id=H5WZYQNHZN2GW3AZISJCYV24RMIWVOMG6PMIJVFKNSKKCFID3P3A, seg_pos={0 0}, upload=471280]: Done uploading piece. err: <nil> Dec 28 00:50:41.877523411 (537h21m41.748606446s) [noise=true , address=77.90.31.13:4629 , piece_id=KYROHTNXD6OZTPYWQFZMWNDJW4WHXCKT6LS4DSFYDEQHPXXNU45A, seg_pos={0 0}, upload=471280]: Done uploading piece. err: <nil> Dec 28 00:50:41.975310333 (537h21m41.846393378s) [noise=true , address=45.91.116.11:4119 , piece_id=QRFSTIPKYGVJ3EEGCK2R4TUWAYGPUZYMUMIFX3446LDECX4MO2KA, seg_pos={0 0}, upload=471280]: Done uploading piece. err: <nil> Dec 28 00:50:41.981593527 (537h21m41.852676562s) [noise=true , address=45.91.116.4:4940 , piece_id=KF245Y4QODMKKKHGFRODVCYT5WIK3JA7DNZBDIYWPC6REKPASIQA, seg_pos={0 0}, upload=471280]: Done uploading piece. err: <nil> Dec 28 00:50:41.990893425 (537h21m41.86197646s) [noise=true , address=45.91.116.6:4974 , piece_id=BCPQNVYSJ5424XKFOAVBBBB3MCOQKLXLRNNGWX4ZVZGIAZPPLL7A, seg_pos={0 0}, upload=471280]: Done uploading piece. err: <nil> Dec 28 00:50:42.001694684 (537h21m41.872777739s) [noise=true , address=145.79.227.41:4086 , piece_id=TF5Q6ZQ2WOL4MT3R4DDQDKY7N6D2E2OGUCGIYHWAKAKZ3TM363LA, seg_pos={0 0}, upload=471280]: Done uploading piece. err: <nil> Dec 28 00:50:42.201683575 (537h21m42.07276661s) [noise=true , address=45.91.116.7:4009 , piece_id=DU3GOV2655OIVVN2KM4CKK2NJSN2UR3KV5SKVOMAOV4UBXU5JTZQ, seg_pos={0 0}, upload=471280]: Done uploading piece. err: <nil> Dec 28 00:50:42.631604345 (537h21m42.50268738s) [noise=true , address=45.91.116.8:4045 , piece_id=BAXTL7SAIBHCFYT2XBFQT5RCHRTDYDC72NMNCXILX7CTY3TZZROA, seg_pos={0 0}, upload=471280]: Done uploading piece. err: <nil> Dec 28 00:50:43.604727923 (537h21m43.475810968s) [noise=true , address=145.79.227.40:4049 , piece_id=BX6R3HBU6CJBLO2O4TGV3OFWJTCPG6EOFLGH2FFN3YG3HALZV23A, seg_pos={0 0}, upload=471280]: Done uploading piece. err: <nil> Dec 28 00:50:43.604741298 (537h21m43.475824343s) [seg_pos={0 0}, upload=471280]: Segment reached optimal threshold of 13 pieces. Dec 28 00:50:43.604969132 (537h21m43.476052167s) [noise=true , address=45.91.116.12:4228 , piece_id=GKF7XDSC6KYCCXRP6IG7KRSDFS76NGFJTSDWG3MTDCLQRQDZZRJA, seg_pos={0 0}, upload=471280]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 28 00:50:43.605009148 (537h21m43.476092183s) [noise=true , address=45.91.116.12:4228 , piece_id=GKF7XDSC6KYCCXRP6IG7KRSDFS76NGFJTSDWG3MTDCLQRQDZZRJA, seg_pos={0 0}, upload=471280]: Long tail cancellation due to optimal threshold Dec 28 00:50:43.605234877 (537h21m43.476317913s) [noise=true , address=77.90.31.14:4664 , piece_id=H4TGPIOVEFLW46ETMDAKCVUSGXQGB4MSKAOYC322IEKZOGLABOOQ, seg_pos={0 0}, upload=471280]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 28 00:50:43.605241200 (537h21m43.476324235s) [noise=true , address=77.90.31.14:4664 , piece_id=H4TGPIOVEFLW46ETMDAKCVUSGXQGB4MSKAOYC322IEKZOGLABOOQ, seg_pos={0 0}, upload=471280]: Long tail cancellation due to optimal threshold Dec 28 00:50:43.605395073 (537h21m43.476478108s) [noise=true , address=145.79.227.48:4303 , piece_id=VDY6RQ6QE7QVWE7NRQXO4Y52QY7H7O477FYYY3WG5AIG5O2H5T2Q, seg_pos={0 0}, upload=471280]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: write tcp 188.166.225.170:40142->145.79.227.48:4303: use of closed network connection; write tcp 188.166.225.170:40142->145.79.227.48:4303: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40142->145.79.227.48:4303: use of closed network connection Dec 28 00:50:43.605400814 (537h21m43.476483859s) [noise=true , address=145.79.227.48:4303 , piece_id=VDY6RQ6QE7QVWE7NRQXO4Y52QY7H7O477FYYY3WG5AIG5O2H5T2Q, seg_pos={0 0}, upload=471280]: Long tail cancellation due to optimal threshold Dec 28 00:50:43.605434969 (537h21m43.476518004s) [seg_pos={0 0}, upload=471280]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:50:43.605525602 (537h21m43.476608647s) [upload=471280]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:50:43.605530972 (537h21m43.476614007s) [upload=471280]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:50:43.902901053 (537h21m43.773984088s) [upload=471280]: Done uploading segments... Dec 28 00:50:43.902948082 (537h21m43.774031118s) [upload=471280]: Upload finished. err: <nil> Duration: 30.002996079s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 42.743820958s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 00:50:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 42.743836347s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 34.605729646s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 00:50:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 34.60574724s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [..X....X.XXX.X.X.XX.X.XXX...X.X.XX.XXX....X...X.X.X...XXXXX.] duration: 8.773545361s trace: https://jaeger.dev.storj.io/trace/70825aec1a198079 FAIL (2025-12-28 00:49:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: HNXCZ7ID5SYJ7N777PJ3JH72AWSHRH7KZQGN5DTNMBKP2UP3U3FQ) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: J6RDFFVCQF6AHCBSXANEC4BSQXT4OADVKDPUP6HKPMPVZ7IH2F6A) context deadline exceeded ======LOGS====== Dec 28 00:48:57.639476352 (537h19m57.510559387s) [upload=471252]: Starting upload Dec 28 00:48:57.639534933 (537h19m57.510617978s) [upload=471252]: Done starting upload Dec 28 00:48:57.639765091 (537h19m57.510848126s) [upload=471252]: Uploading segments... Dec 28 00:48:57.639802913 (537h19m57.510885948s) [upload=471252]: Got next segment. Inline: false Dec 28 00:48:57.639827681 (537h19m57.510910716s) [upload=471252]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:48:57.639832890 (537h19m57.510915926s) [upload=471252]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:48:57.887289909 (537h19m57.758372964s) [seg_pos={0 0}, upload=471252]: Begin upload segment... Dec 28 00:48:57.887367186 (537h19m57.758450221s) [seg_pos={0 0}, upload=471252]: Done begin upload segment. Dec 28 00:48:57.887449111 (537h19m57.758532156s) [noise=true , address=145.79.227.36:4518 , piece_id=IR2KAB7XEGQHBBGPTLO6FO5SN4MSV3QGEQELA7WRT3OVP6WULR6Q, seg_pos={0 0}, upload=471252]: Uploading piece... Dec 28 00:48:57.887525808 (537h19m57.758608853s) [noise=true , address=145.79.227.30:4762 , piece_id=DXSQKCSMJYLDJQZSFOYDVFKFFJPHCIZUYZ3OLIUZDZCUOM76K4ZQ, seg_pos={0 0}, upload=471252]: Uploading piece... Dec 28 00:48:57.888275285 (537h19m57.75935832s) [noise=true , address=45.91.116.8:4046 , piece_id=HVO45UL54WKCR5DBCH2ZMEEOKCZ47HVOTYZOY77YYTKZQZZSSYRA, seg_pos={0 0}, upload=471252]: Uploading piece... Dec 28 00:48:57.888414029 (537h19m57.759497064s) [noise=true , address=145.79.227.46:4602 , piece_id=F732IPXW2VF2K5YIX4J4RIULZ4QI7WMFFYGRZEUP5GANRR7S3NSA, seg_pos={0 0}, upload=471252]: Uploading piece... Dec 28 00:48:57.888756551 (537h19m57.759839586s) [noise=true , address=45.91.116.14:4192 , piece_id=HNXCZ7ID5SYJ7N777PJ3JH72AWSHRH7KZQGN5DTNMBKP2UP3U3FQ, seg_pos={0 0}, upload=471252]: Uploading piece... Dec 28 00:48:57.888890607 (537h19m57.759973642s) [noise=true , address=45.91.116.11:4118 , piece_id=HLIQE3HDNHM3JHNZ7E3HU76TNVTOG53HYQBXVL7C4VP2KPZQR7UA, seg_pos={0 0}, upload=471252]: Uploading piece... Dec 28 00:48:57.889207921 (537h19m57.760290966s) [noise=true , address=45.91.116.9:4263 , piece_id=ZHKDOCAFOHLXSJNMP4ZFR4ROUTZYPW5NSDPR7MI6QX6JYWRICXVQ, seg_pos={0 0}, upload=471252]: Uploading piece... Dec 28 00:48:57.889352266 (537h19m57.760435311s) [noise=true , address=145.79.227.48:4303 , piece_id=WBJ3X6S2KCSI2QULYE2QT7V6VCU7RPERU6QJP5NW275GYD4IDFVA, seg_pos={0 0}, upload=471252]: Uploading piece... Dec 28 00:48:57.889616589 (537h19m57.760699624s) [noise=true , address=145.79.227.9:4365 , piece_id=6BFAPKBNJX4MUHNGGDJ3FSVGPB75JFEF7FYQKKZGBXLFBVZYF4MQ, seg_pos={0 0}, upload=471252]: Uploading piece... Dec 28 00:48:57.889697573 (537h19m57.760780618s) [noise=true , address=145.79.227.51:4409 , piece_id=SLGLZV6TVEFX6742JZPW3V7EAXYCYGCFADKORTEYEGDDMIWLBZKA, seg_pos={0 0}, upload=471252]: Uploading piece... Dec 28 00:48:57.890085722 (537h19m57.761168757s) [noise=true , address=45.91.116.13:4154 , piece_id=J6RDFFVCQF6AHCBSXANEC4BSQXT4OADVKDPUP6HKPMPVZ7IH2F6A, seg_pos={0 0}, upload=471252]: Uploading piece... Dec 28 00:48:57.890104358 (537h19m57.761187393s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSQKPK7AFUT5QXYIY2TT3NYV5Y6ZGK5MK5CYGOAQKIU22VMM3PDQ, seg_pos={0 0}, upload=471252]: Uploading piece... Dec 28 00:48:57.890563051 (537h19m57.761646086s) [noise=true , address=145.79.227.37:4566 , piece_id=JBG5SR2NRGVSGVIMKFKHDFHZTEQ6L5F2OX3IFYVW7A3T2GD4GKUA, seg_pos={0 0}, upload=471252]: Uploading piece... Dec 28 00:48:57.890636501 (537h19m57.761719546s) [noise=true , address=145.79.227.52:4446 , piece_id=4N56DPM4QAWR2LPSUYGA52BDC4N64AXE4LHQIEJCLA55A3VLZNJQ, seg_pos={0 0}, upload=471252]: Uploading piece... Dec 28 00:48:57.890907688 (537h19m57.761990723s) [noise=true , address=145.79.227.31:4798 , piece_id=WSBA2AJTNBQXSPUBRYE75OOCXUWFCNUGS3QJ7GLGWE6Y33CET4UA, seg_pos={0 0}, upload=471252]: Uploading piece... Dec 28 00:48:57.891042144 (537h19m57.762125179s) [noise=true , address=145.79.227.2:4115 , piece_id=US2LMTJ3LF6WUYRERHNRQ2CX2X3COR3Y3BZBFJT3KV6RYPRK2KQA, seg_pos={0 0}, upload=471252]: Uploading piece... Dec 28 00:48:59.598550709 (537h19m59.469633754s) [upload=471252]: Next returned nil segment Dec 28 00:48:59.598604320 (537h19m59.469687365s) [upload=471252]: Waiting for error group managing segments... Dec 28 00:48:59.890084687 (537h19m59.761167742s) [noise=true , address=145.79.227.51:4409 , piece_id=SLGLZV6TVEFX6742JZPW3V7EAXYCYGCFADKORTEYEGDDMIWLBZKA, seg_pos={0 0}, upload=471252]: Done uploading piece. err: <nil> Dec 28 00:48:59.894586009 (537h19m59.765669054s) [noise=true , address=145.79.227.36:4518 , piece_id=IR2KAB7XEGQHBBGPTLO6FO5SN4MSV3QGEQELA7WRT3OVP6WULR6Q, seg_pos={0 0}, upload=471252]: Done uploading piece. err: <nil> Dec 28 00:48:59.895432190 (537h19m59.766515235s) [noise=true , address=145.79.227.31:4798 , piece_id=WSBA2AJTNBQXSPUBRYE75OOCXUWFCNUGS3QJ7GLGWE6Y33CET4UA, seg_pos={0 0}, upload=471252]: Done uploading piece. err: <nil> Dec 28 00:48:59.896245268 (537h19m59.767328303s) [noise=true , address=145.79.227.52:4446 , piece_id=4N56DPM4QAWR2LPSUYGA52BDC4N64AXE4LHQIEJCLA55A3VLZNJQ, seg_pos={0 0}, upload=471252]: Done uploading piece. err: <nil> Dec 28 00:48:59.898989013 (537h19m59.770072048s) [noise=true , address=145.79.227.9:4365 , piece_id=6BFAPKBNJX4MUHNGGDJ3FSVGPB75JFEF7FYQKKZGBXLFBVZYF4MQ, seg_pos={0 0}, upload=471252]: Done uploading piece. err: <nil> Dec 28 00:48:59.905620701 (537h19m59.776703736s) [noise=true , address=145.79.227.46:4602 , piece_id=F732IPXW2VF2K5YIX4J4RIULZ4QI7WMFFYGRZEUP5GANRR7S3NSA, seg_pos={0 0}, upload=471252]: Done uploading piece. err: <nil> Dec 28 00:48:59.917306672 (537h19m59.788389717s) [noise=true , address=145.79.227.48:4303 , piece_id=WBJ3X6S2KCSI2QULYE2QT7V6VCU7RPERU6QJP5NW275GYD4IDFVA, seg_pos={0 0}, upload=471252]: Done uploading piece. err: <nil> Dec 28 00:48:59.920791368 (537h19m59.791874413s) [noise=true , address=145.79.227.2:4115 , piece_id=US2LMTJ3LF6WUYRERHNRQ2CX2X3COR3Y3BZBFJT3KV6RYPRK2KQA, seg_pos={0 0}, upload=471252]: Done uploading piece. err: <nil> Dec 28 00:49:00.164390350 (537h20m0.035473395s) [noise=true , address=45.91.116.11:4118 , piece_id=HLIQE3HDNHM3JHNZ7E3HU76TNVTOG53HYQBXVL7C4VP2KPZQR7UA, seg_pos={0 0}, upload=471252]: Done uploading piece. err: <nil> Dec 28 00:49:00.176382385 (537h20m0.04746543s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSQKPK7AFUT5QXYIY2TT3NYV5Y6ZGK5MK5CYGOAQKIU22VMM3PDQ, seg_pos={0 0}, upload=471252]: Done uploading piece. err: <nil> Dec 28 00:49:00.176575122 (537h20m0.047658167s) [noise=true , address=45.91.116.14:4192 , piece_id=HNXCZ7ID5SYJ7N777PJ3JH72AWSHRH7KZQGN5DTNMBKP2UP3U3FQ, seg_pos={0 0}, upload=471252]: Done uploading piece. err: <nil> Dec 28 00:49:00.197047054 (537h20m0.068130099s) [noise=true , address=45.91.116.13:4154 , piece_id=J6RDFFVCQF6AHCBSXANEC4BSQXT4OADVKDPUP6HKPMPVZ7IH2F6A, seg_pos={0 0}, upload=471252]: Done uploading piece. err: <nil> Dec 28 00:49:00.242705114 (537h20m0.11378815s) [noise=true , address=45.91.116.9:4263 , piece_id=ZHKDOCAFOHLXSJNMP4ZFR4ROUTZYPW5NSDPR7MI6QX6JYWRICXVQ, seg_pos={0 0}, upload=471252]: Done uploading piece. err: <nil> Dec 28 00:49:00.242724762 (537h20m0.113807807s) [seg_pos={0 0}, upload=471252]: Segment reached optimal threshold of 13 pieces. Dec 28 00:49:00.243118271 (537h20m0.114201306s) [noise=true , address=45.91.116.8:4046 , piece_id=HVO45UL54WKCR5DBCH2ZMEEOKCZ47HVOTYZOY77YYTKZQZZSSYRA, seg_pos={0 0}, upload=471252]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 28 00:49:00.243130044 (537h20m0.11421309s) [noise=true , address=45.91.116.8:4046 , piece_id=HVO45UL54WKCR5DBCH2ZMEEOKCZ47HVOTYZOY77YYTKZQZZSSYRA, seg_pos={0 0}, upload=471252]: Long tail cancellation due to optimal threshold Dec 28 00:49:00.244128535 (537h20m0.11521157s) [noise=true , address=145.79.227.37:4566 , piece_id=JBG5SR2NRGVSGVIMKFKHDFHZTEQ6L5F2OX3IFYVW7A3T2GD4GKUA, seg_pos={0 0}, upload=471252]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 28 00:49:00.244142212 (537h20m0.115225247s) [noise=true , address=145.79.227.37:4566 , piece_id=JBG5SR2NRGVSGVIMKFKHDFHZTEQ6L5F2OX3IFYVW7A3T2GD4GKUA, seg_pos={0 0}, upload=471252]: Long tail cancellation due to optimal threshold Dec 28 00:49:00.244531363 (537h20m0.115614398s) [noise=true , address=145.79.227.30:4762 , piece_id=DXSQKCSMJYLDJQZSFOYDVFKFFJPHCIZUYZ3OLIUZDZCUOM76K4ZQ, seg_pos={0 0}, upload=471252]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 28 00:49:00.244537955 (537h20m0.11562099s) [noise=true , address=145.79.227.30:4762 , piece_id=DXSQKCSMJYLDJQZSFOYDVFKFFJPHCIZUYZ3OLIUZDZCUOM76K4ZQ, seg_pos={0 0}, upload=471252]: Long tail cancellation due to optimal threshold Dec 28 00:49:00.244581488 (537h20m0.115664523s) [seg_pos={0 0}, upload=471252]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:49:00.244670357 (537h20m0.115753392s) [upload=471252]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:49:00.244676178 (537h20m0.115759213s) [upload=471252]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:49:00.537079852 (537h20m0.408162917s) [upload=471252]: Done uploading segments... Dec 28 00:49:00.537153372 (537h20m0.408236417s) [upload=471252]: Upload finished. err: <nil> Duration: 8.773577843s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 34.951792422s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 00:49:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 34.951824834s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [...X....X.XXX.X.X.XX.X.XXX...X.X.XX.XXX....X...X.X.X...XXXXX] duration: 30.002886692s trace: https://jaeger.dev.storj.io/trace/70825aec1a198079 FAIL (2025-12-28 00:49:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: HNXCZ7ID5SYJ7N777PJ3JH72AWSHRH7KZQGN5DTNMBKP2UP3U3FQ) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: J6RDFFVCQF6AHCBSXANEC4BSQXT4OADVKDPUP6HKPMPVZ7IH2F6A) context deadline exceeded ======LOGS====== Dec 28 00:48:57.639476352 (537h19m57.510559387s) [upload=471252]: Starting upload Dec 28 00:48:57.639534933 (537h19m57.510617978s) [upload=471252]: Done starting upload Dec 28 00:48:57.639765091 (537h19m57.510848126s) [upload=471252]: Uploading segments... Dec 28 00:48:57.639802913 (537h19m57.510885948s) [upload=471252]: Got next segment. Inline: false Dec 28 00:48:57.639827681 (537h19m57.510910716s) [upload=471252]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:48:57.639832890 (537h19m57.510915926s) [upload=471252]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:48:57.887289909 (537h19m57.758372964s) [seg_pos={0 0}, upload=471252]: Begin upload segment... Dec 28 00:48:57.887367186 (537h19m57.758450221s) [seg_pos={0 0}, upload=471252]: Done begin upload segment. Dec 28 00:48:57.887449111 (537h19m57.758532156s) [noise=true , address=145.79.227.36:4518 , piece_id=IR2KAB7XEGQHBBGPTLO6FO5SN4MSV3QGEQELA7WRT3OVP6WULR6Q, seg_pos={0 0}, upload=471252]: Uploading piece... Dec 28 00:48:57.887525808 (537h19m57.758608853s) [noise=true , address=145.79.227.30:4762 , piece_id=DXSQKCSMJYLDJQZSFOYDVFKFFJPHCIZUYZ3OLIUZDZCUOM76K4ZQ, seg_pos={0 0}, upload=471252]: Uploading piece... Dec 28 00:48:57.888275285 (537h19m57.75935832s) [noise=true , address=45.91.116.8:4046 , piece_id=HVO45UL54WKCR5DBCH2ZMEEOKCZ47HVOTYZOY77YYTKZQZZSSYRA, seg_pos={0 0}, upload=471252]: Uploading piece... Dec 28 00:48:57.888414029 (537h19m57.759497064s) [noise=true , address=145.79.227.46:4602 , piece_id=F732IPXW2VF2K5YIX4J4RIULZ4QI7WMFFYGRZEUP5GANRR7S3NSA, seg_pos={0 0}, upload=471252]: Uploading piece... Dec 28 00:48:57.888756551 (537h19m57.759839586s) [noise=true , address=45.91.116.14:4192 , piece_id=HNXCZ7ID5SYJ7N777PJ3JH72AWSHRH7KZQGN5DTNMBKP2UP3U3FQ, seg_pos={0 0}, upload=471252]: Uploading piece... Dec 28 00:48:57.888890607 (537h19m57.759973642s) [noise=true , address=45.91.116.11:4118 , piece_id=HLIQE3HDNHM3JHNZ7E3HU76TNVTOG53HYQBXVL7C4VP2KPZQR7UA, seg_pos={0 0}, upload=471252]: Uploading piece... Dec 28 00:48:57.889207921 (537h19m57.760290966s) [noise=true , address=45.91.116.9:4263 , piece_id=ZHKDOCAFOHLXSJNMP4ZFR4ROUTZYPW5NSDPR7MI6QX6JYWRICXVQ, seg_pos={0 0}, upload=471252]: Uploading piece... Dec 28 00:48:57.889352266 (537h19m57.760435311s) [noise=true , address=145.79.227.48:4303 , piece_id=WBJ3X6S2KCSI2QULYE2QT7V6VCU7RPERU6QJP5NW275GYD4IDFVA, seg_pos={0 0}, upload=471252]: Uploading piece... Dec 28 00:48:57.889616589 (537h19m57.760699624s) [noise=true , address=145.79.227.9:4365 , piece_id=6BFAPKBNJX4MUHNGGDJ3FSVGPB75JFEF7FYQKKZGBXLFBVZYF4MQ, seg_pos={0 0}, upload=471252]: Uploading piece... Dec 28 00:48:57.889697573 (537h19m57.760780618s) [noise=true , address=145.79.227.51:4409 , piece_id=SLGLZV6TVEFX6742JZPW3V7EAXYCYGCFADKORTEYEGDDMIWLBZKA, seg_pos={0 0}, upload=471252]: Uploading piece... Dec 28 00:48:57.890085722 (537h19m57.761168757s) [noise=true , address=45.91.116.13:4154 , piece_id=J6RDFFVCQF6AHCBSXANEC4BSQXT4OADVKDPUP6HKPMPVZ7IH2F6A, seg_pos={0 0}, upload=471252]: Uploading piece... Dec 28 00:48:57.890104358 (537h19m57.761187393s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSQKPK7AFUT5QXYIY2TT3NYV5Y6ZGK5MK5CYGOAQKIU22VMM3PDQ, seg_pos={0 0}, upload=471252]: Uploading piece... Dec 28 00:48:57.890563051 (537h19m57.761646086s) [noise=true , address=145.79.227.37:4566 , piece_id=JBG5SR2NRGVSGVIMKFKHDFHZTEQ6L5F2OX3IFYVW7A3T2GD4GKUA, seg_pos={0 0}, upload=471252]: Uploading piece... Dec 28 00:48:57.890636501 (537h19m57.761719546s) [noise=true , address=145.79.227.52:4446 , piece_id=4N56DPM4QAWR2LPSUYGA52BDC4N64AXE4LHQIEJCLA55A3VLZNJQ, seg_pos={0 0}, upload=471252]: Uploading piece... Dec 28 00:48:57.890907688 (537h19m57.761990723s) [noise=true , address=145.79.227.31:4798 , piece_id=WSBA2AJTNBQXSPUBRYE75OOCXUWFCNUGS3QJ7GLGWE6Y33CET4UA, seg_pos={0 0}, upload=471252]: Uploading piece... Dec 28 00:48:57.891042144 (537h19m57.762125179s) [noise=true , address=145.79.227.2:4115 , piece_id=US2LMTJ3LF6WUYRERHNRQ2CX2X3COR3Y3BZBFJT3KV6RYPRK2KQA, seg_pos={0 0}, upload=471252]: Uploading piece... Dec 28 00:48:59.598550709 (537h19m59.469633754s) [upload=471252]: Next returned nil segment Dec 28 00:48:59.598604320 (537h19m59.469687365s) [upload=471252]: Waiting for error group managing segments... Dec 28 00:48:59.890084687 (537h19m59.761167742s) [noise=true , address=145.79.227.51:4409 , piece_id=SLGLZV6TVEFX6742JZPW3V7EAXYCYGCFADKORTEYEGDDMIWLBZKA, seg_pos={0 0}, upload=471252]: Done uploading piece. err: <nil> Dec 28 00:48:59.894586009 (537h19m59.765669054s) [noise=true , address=145.79.227.36:4518 , piece_id=IR2KAB7XEGQHBBGPTLO6FO5SN4MSV3QGEQELA7WRT3OVP6WULR6Q, seg_pos={0 0}, upload=471252]: Done uploading piece. err: <nil> Dec 28 00:48:59.895432190 (537h19m59.766515235s) [noise=true , address=145.79.227.31:4798 , piece_id=WSBA2AJTNBQXSPUBRYE75OOCXUWFCNUGS3QJ7GLGWE6Y33CET4UA, seg_pos={0 0}, upload=471252]: Done uploading piece. err: <nil> Dec 28 00:48:59.896245268 (537h19m59.767328303s) [noise=true , address=145.79.227.52:4446 , piece_id=4N56DPM4QAWR2LPSUYGA52BDC4N64AXE4LHQIEJCLA55A3VLZNJQ, seg_pos={0 0}, upload=471252]: Done uploading piece. err: <nil> Dec 28 00:48:59.898989013 (537h19m59.770072048s) [noise=true , address=145.79.227.9:4365 , piece_id=6BFAPKBNJX4MUHNGGDJ3FSVGPB75JFEF7FYQKKZGBXLFBVZYF4MQ, seg_pos={0 0}, upload=471252]: Done uploading piece. err: <nil> Dec 28 00:48:59.905620701 (537h19m59.776703736s) [noise=true , address=145.79.227.46:4602 , piece_id=F732IPXW2VF2K5YIX4J4RIULZ4QI7WMFFYGRZEUP5GANRR7S3NSA, seg_pos={0 0}, upload=471252]: Done uploading piece. err: <nil> Dec 28 00:48:59.917306672 (537h19m59.788389717s) [noise=true , address=145.79.227.48:4303 , piece_id=WBJ3X6S2KCSI2QULYE2QT7V6VCU7RPERU6QJP5NW275GYD4IDFVA, seg_pos={0 0}, upload=471252]: Done uploading piece. err: <nil> Dec 28 00:48:59.920791368 (537h19m59.791874413s) [noise=true , address=145.79.227.2:4115 , piece_id=US2LMTJ3LF6WUYRERHNRQ2CX2X3COR3Y3BZBFJT3KV6RYPRK2KQA, seg_pos={0 0}, upload=471252]: Done uploading piece. err: <nil> Dec 28 00:49:00.164390350 (537h20m0.035473395s) [noise=true , address=45.91.116.11:4118 , piece_id=HLIQE3HDNHM3JHNZ7E3HU76TNVTOG53HYQBXVL7C4VP2KPZQR7UA, seg_pos={0 0}, upload=471252]: Done uploading piece. err: <nil> Dec 28 00:49:00.176382385 (537h20m0.04746543s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSQKPK7AFUT5QXYIY2TT3NYV5Y6ZGK5MK5CYGOAQKIU22VMM3PDQ, seg_pos={0 0}, upload=471252]: Done uploading piece. err: <nil> Dec 28 00:49:00.176575122 (537h20m0.047658167s) [noise=true , address=45.91.116.14:4192 , piece_id=HNXCZ7ID5SYJ7N777PJ3JH72AWSHRH7KZQGN5DTNMBKP2UP3U3FQ, seg_pos={0 0}, upload=471252]: Done uploading piece. err: <nil> Dec 28 00:49:00.197047054 (537h20m0.068130099s) [noise=true , address=45.91.116.13:4154 , piece_id=J6RDFFVCQF6AHCBSXANEC4BSQXT4OADVKDPUP6HKPMPVZ7IH2F6A, seg_pos={0 0}, upload=471252]: Done uploading piece. err: <nil> Dec 28 00:49:00.242705114 (537h20m0.11378815s) [noise=true , address=45.91.116.9:4263 , piece_id=ZHKDOCAFOHLXSJNMP4ZFR4ROUTZYPW5NSDPR7MI6QX6JYWRICXVQ, seg_pos={0 0}, upload=471252]: Done uploading piece. err: <nil> Dec 28 00:49:00.242724762 (537h20m0.113807807s) [seg_pos={0 0}, upload=471252]: Segment reached optimal threshold of 13 pieces. Dec 28 00:49:00.243118271 (537h20m0.114201306s) [noise=true , address=45.91.116.8:4046 , piece_id=HVO45UL54WKCR5DBCH2ZMEEOKCZ47HVOTYZOY77YYTKZQZZSSYRA, seg_pos={0 0}, upload=471252]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 28 00:49:00.243130044 (537h20m0.11421309s) [noise=true , address=45.91.116.8:4046 , piece_id=HVO45UL54WKCR5DBCH2ZMEEOKCZ47HVOTYZOY77YYTKZQZZSSYRA, seg_pos={0 0}, upload=471252]: Long tail cancellation due to optimal threshold Dec 28 00:49:00.244128535 (537h20m0.11521157s) [noise=true , address=145.79.227.37:4566 , piece_id=JBG5SR2NRGVSGVIMKFKHDFHZTEQ6L5F2OX3IFYVW7A3T2GD4GKUA, seg_pos={0 0}, upload=471252]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SaNJZaVix7BSbEWhu2Vmt3XjDDWCsQ8wTSiFR7MQ8oGwD9N2F): protocol: expected piece hash; context canceled; EOF Dec 28 00:49:00.244142212 (537h20m0.115225247s) [noise=true , address=145.79.227.37:4566 , piece_id=JBG5SR2NRGVSGVIMKFKHDFHZTEQ6L5F2OX3IFYVW7A3T2GD4GKUA, seg_pos={0 0}, upload=471252]: Long tail cancellation due to optimal threshold Dec 28 00:49:00.244531363 (537h20m0.115614398s) [noise=true , address=145.79.227.30:4762 , piece_id=DXSQKCSMJYLDJQZSFOYDVFKFFJPHCIZUYZ3OLIUZDZCUOM76K4ZQ, seg_pos={0 0}, upload=471252]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 28 00:49:00.244537955 (537h20m0.11562099s) [noise=true , address=145.79.227.30:4762 , piece_id=DXSQKCSMJYLDJQZSFOYDVFKFFJPHCIZUYZ3OLIUZDZCUOM76K4ZQ, seg_pos={0 0}, upload=471252]: Long tail cancellation due to optimal threshold Dec 28 00:49:00.244581488 (537h20m0.115664523s) [seg_pos={0 0}, upload=471252]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:49:00.244670357 (537h20m0.115753392s) [upload=471252]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:49:00.244676178 (537h20m0.115759213s) [upload=471252]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:49:00.537079852 (537h20m0.408162917s) [upload=471252]: Done uploading segments... Dec 28 00:49:00.537153372 (537h20m0.408236417s) [upload=471252]: Upload finished. err: <nil> Duration: 30.002897833s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [....X....X.XXX.X.X.XX.X.XXX...X.X.XX.XXX....X...X.X.X...XXXX] duration: 30.003238234s trace: https://jaeger.dev.storj.io/trace/42ec53f61b416fb2 FAIL (2025-12-28 00:48:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 00:48:27.447158880 (537h19m27.318241915s) [upload=471240]: Starting upload Dec 28 00:48:27.447217212 (537h19m27.318300247s) [upload=471240]: Done starting upload Dec 28 00:48:27.447271144 (537h19m27.318354189s) [upload=471240]: Uploading segments... Dec 28 00:48:27.447401232 (537h19m27.318484267s) [upload=471240]: Got next segment. Inline: false Dec 28 00:48:27.447433884 (537h19m27.318516929s) [upload=471240]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:48:27.447439775 (537h19m27.31852282s) [upload=471240]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:48:27.694016928 (537h19m27.565099963s) [seg_pos={0 0}, upload=471240]: Begin upload segment... Dec 28 00:48:27.694078094 (537h19m27.565161139s) [seg_pos={0 0}, upload=471240]: Done begin upload segment. Dec 28 00:48:27.694129823 (537h19m27.565212858s) [noise=true , address=45.91.116.7:4011 , piece_id=CBDSG7P32QXLKFGU7SH5FNL2FMQEG2JODEZCEWJFFPQGTCZKTHUQ, seg_pos={0 0}, upload=471240]: Uploading piece... Dec 28 00:48:27.694139381 (537h19m27.565222416s) [noise=true , address=45.91.116.4:4939 , piece_id=STTB2Y3VA455EE3NOUIFOPE74NNP6GEHAOJB5KIZ6GN54QIQDTNQ, seg_pos={0 0}, upload=471240]: Uploading piece... Dec 28 00:48:27.694658219 (537h19m27.565741254s) [noise=true , address=145.79.227.15:4874 , piece_id=NCVHOHHN4CNH3FSPI4JGAP5LJSSCZVTGTEZVIPJY4CAFBRGYMBWA, seg_pos={0 0}, upload=471240]: Uploading piece... Dec 28 00:48:27.695053301 (537h19m27.566136346s) [noise=true , address=45.91.116.5:4300 , piece_id=DAMXVBODNKRD5DXQU2OS5VIPIGKLPJM75PUC2DGZQP4GMZLZOCIA, seg_pos={0 0}, upload=471240]: Uploading piece... Dec 28 00:48:27.695462791 (537h19m27.566545836s) [noise=true , address=145.79.227.51:4409 , piece_id=MPDGF5S2WS6KOYH4BPHTKHJIMWNDUQAX4I3WCTPSLOOZSW4NQLOA, seg_pos={0 0}, upload=471240]: Uploading piece... Dec 28 00:48:27.695472559 (537h19m27.566555604s) [noise=true , address=145.79.227.45:4230 , piece_id=LXCKNYGX55CMGJRVFA6AFLIJQT2VEK7KACFSXR3VINGBTEE556IA, seg_pos={0 0}, upload=471240]: Uploading piece... Dec 28 00:48:27.695763093 (537h19m27.566846138s) [noise=true , address=45.91.116.9:4262 , piece_id=RZAKGMJUNXNPHKDPSGTBVOFQ6AS6NBCQOC3MFWQ4JIK5LXQ4QXPQ, seg_pos={0 0}, upload=471240]: Uploading piece... Dec 28 00:48:27.696033186 (537h19m27.567116232s) [noise=true , address=145.79.227.48:4301 , piece_id=YKU5Y7GXLB5HHCX5L3IRUOTONQNN3UWBYUE54DM3OJUKEJG3CJVA, seg_pos={0 0}, upload=471240]: Uploading piece... Dec 28 00:48:27.696412840 (537h19m27.567495875s) [noise=true , address=45.91.116.10:4083 , piece_id=JU5SEADP7VYRMKFH5SV7SUOFDBY7E76BSXNJTNPPGJJNQIXLDXGA, seg_pos={0 0}, upload=471240]: Uploading piece... Dec 28 00:48:27.696763026 (537h19m27.567846071s) [noise=true , address=45.91.116.14:4189 , piece_id=UWTBBQ6Y7UKPQ365HO2JLJHBWYH3MB7EAUSVG22SGL2DTHLY4GNA, seg_pos={0 0}, upload=471240]: Uploading piece... Dec 28 00:48:27.697105649 (537h19m27.568188684s) [noise=true , address=45.91.116.8:4046 , piece_id=TOUAQFE5ZQ2LA34D35MO4BVDY2SEKDNVJVDO5LN3TJFBNLJNP5OQ, seg_pos={0 0}, upload=471240]: Uploading piece... Dec 28 00:48:27.697473710 (537h19m27.568556755s) [noise=true , address=145.79.227.4:4185 , piece_id=PEQA2OIQ232CFPHA35KMHYBHU7HIHML7OTU6OTF76DS37VPBJQQQ, seg_pos={0 0}, upload=471240]: Uploading piece... Dec 28 00:48:27.697901484 (537h19m27.568984529s) [noise=true , address=77.90.31.11:4556 , piece_id=N4S4ONB67O2SF3UMVR3RQXLWDLFS3YMCONMRH4ZSC6J3YAWH46PA, seg_pos={0 0}, upload=471240]: Uploading piece... Dec 28 00:48:27.704403435 (537h19m27.57548647s) [noise=true , address=45.91.116.11:4120 , piece_id=TERMUEIIEHBLSYREL7TGQRCMJTHWLSS4N4QJXYYMPXWMCSDQS36A, seg_pos={0 0}, upload=471240]: Uploading piece... Dec 28 00:48:27.713359629 (537h19m27.584442674s) [noise=true , address=145.79.227.2:4114 , piece_id=LRCRYW5CC6WDNUASTQGIL4JTHWYH6ERKOS6UYDHCZY2EBP3UGMCA, seg_pos={0 0}, upload=471240]: Uploading piece... Dec 28 00:48:27.717187098 (537h19m27.588270133s) [noise=true , address=145.79.227.41:4087 , piece_id=OVCHL27GIH7IY2KV7GKFJMB5GJKTTS5DS6ECWG3WF5JJPZFGVAKQ, seg_pos={0 0}, upload=471240]: Uploading piece... Dec 28 00:48:29.407909066 (537h19m29.278992101s) [upload=471240]: Next returned nil segment Dec 28 00:48:29.407921439 (537h19m29.279004484s) [upload=471240]: Waiting for error group managing segments... Dec 28 00:48:29.682955298 (537h19m29.554038333s) [noise=true , address=145.79.227.15:4874 , piece_id=NCVHOHHN4CNH3FSPI4JGAP5LJSSCZVTGTEZVIPJY4CAFBRGYMBWA, seg_pos={0 0}, upload=471240]: Done uploading piece. err: <nil> Dec 28 00:48:29.706611062 (537h19m29.577694108s) [noise=true , address=145.79.227.45:4230 , piece_id=LXCKNYGX55CMGJRVFA6AFLIJQT2VEK7KACFSXR3VINGBTEE556IA, seg_pos={0 0}, upload=471240]: Done uploading piece. err: <nil> Dec 28 00:48:29.710539764 (537h19m29.581622799s) [noise=true , address=145.79.227.48:4301 , piece_id=YKU5Y7GXLB5HHCX5L3IRUOTONQNN3UWBYUE54DM3OJUKEJG3CJVA, seg_pos={0 0}, upload=471240]: Done uploading piece. err: <nil> Dec 28 00:48:29.718774945 (537h19m29.58985798s) [noise=true , address=145.79.227.41:4087 , piece_id=OVCHL27GIH7IY2KV7GKFJMB5GJKTTS5DS6ECWG3WF5JJPZFGVAKQ, seg_pos={0 0}, upload=471240]: Done uploading piece. err: <nil> Dec 28 00:48:29.720001000 (537h19m29.591084035s) [noise=true , address=145.79.227.51:4409 , piece_id=MPDGF5S2WS6KOYH4BPHTKHJIMWNDUQAX4I3WCTPSLOOZSW4NQLOA, seg_pos={0 0}, upload=471240]: Done uploading piece. err: <nil> Dec 28 00:48:29.726573585 (537h19m29.59765662s) [noise=true , address=145.79.227.4:4185 , piece_id=PEQA2OIQ232CFPHA35KMHYBHU7HIHML7OTU6OTF76DS37VPBJQQQ, seg_pos={0 0}, upload=471240]: Done uploading piece. err: <nil> Dec 28 00:48:29.727821841 (537h19m29.598904886s) [noise=true , address=145.79.227.2:4114 , piece_id=LRCRYW5CC6WDNUASTQGIL4JTHWYH6ERKOS6UYDHCZY2EBP3UGMCA, seg_pos={0 0}, upload=471240]: Done uploading piece. err: <nil> Dec 28 00:48:29.975147250 (537h19m29.846230285s) [noise=true , address=45.91.116.11:4120 , piece_id=TERMUEIIEHBLSYREL7TGQRCMJTHWLSS4N4QJXYYMPXWMCSDQS36A, seg_pos={0 0}, upload=471240]: Done uploading piece. err: <nil> Dec 28 00:48:29.984466284 (537h19m29.855549329s) [noise=true , address=45.91.116.8:4046 , piece_id=TOUAQFE5ZQ2LA34D35MO4BVDY2SEKDNVJVDO5LN3TJFBNLJNP5OQ, seg_pos={0 0}, upload=471240]: Done uploading piece. err: <nil> Dec 28 00:48:30.057922748 (537h19m29.929005783s) [noise=true , address=45.91.116.7:4011 , piece_id=CBDSG7P32QXLKFGU7SH5FNL2FMQEG2JODEZCEWJFFPQGTCZKTHUQ, seg_pos={0 0}, upload=471240]: Done uploading piece. err: <nil> Dec 28 00:48:30.060041411 (537h19m29.931124446s) [noise=true , address=45.91.116.10:4083 , piece_id=JU5SEADP7VYRMKFH5SV7SUOFDBY7E76BSXNJTNPPGJJNQIXLDXGA, seg_pos={0 0}, upload=471240]: Done uploading piece. err: <nil> Dec 28 00:48:30.060274615 (537h19m29.93135765s) [noise=true , address=45.91.116.4:4939 , piece_id=STTB2Y3VA455EE3NOUIFOPE74NNP6GEHAOJB5KIZ6GN54QIQDTNQ, seg_pos={0 0}, upload=471240]: Done uploading piece. err: <nil> Dec 28 00:48:30.095173248 (537h19m29.966256293s) [noise=true , address=45.91.116.5:4300 , piece_id=DAMXVBODNKRD5DXQU2OS5VIPIGKLPJM75PUC2DGZQP4GMZLZOCIA, seg_pos={0 0}, upload=471240]: Done uploading piece. err: <nil> Dec 28 00:48:30.095192585 (537h19m29.96627563s) [seg_pos={0 0}, upload=471240]: Segment reached optimal threshold of 13 pieces. Dec 28 00:48:30.095366586 (537h19m29.966449632s) [noise=true , address=77.90.31.11:4556 , piece_id=N4S4ONB67O2SF3UMVR3RQXLWDLFS3YMCONMRH4ZSC6J3YAWH46PA, seg_pos={0 0}, upload=471240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 28 00:48:30.095372578 (537h19m29.966455613s) [noise=true , address=77.90.31.11:4556 , piece_id=N4S4ONB67O2SF3UMVR3RQXLWDLFS3YMCONMRH4ZSC6J3YAWH46PA, seg_pos={0 0}, upload=471240]: Long tail cancellation due to optimal threshold Dec 28 00:48:30.095480774 (537h19m29.966563809s) [noise=true , address=45.91.116.9:4262 , piece_id=RZAKGMJUNXNPHKDPSGTBVOFQ6AS6NBCQOC3MFWQ4JIK5LXQ4QXPQ, seg_pos={0 0}, upload=471240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 28 00:48:30.095488678 (537h19m29.966571723s) [noise=true , address=45.91.116.9:4262 , piece_id=RZAKGMJUNXNPHKDPSGTBVOFQ6AS6NBCQOC3MFWQ4JIK5LXQ4QXPQ, seg_pos={0 0}, upload=471240]: Long tail cancellation due to optimal threshold Dec 28 00:48:30.095577278 (537h19m29.966660323s) [noise=true , address=45.91.116.14:4189 , piece_id=UWTBBQ6Y7UKPQ365HO2JLJHBWYH3MB7EAUSVG22SGL2DTHLY4GNA, seg_pos={0 0}, upload=471240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:60946->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:60946->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60946->45.91.116.14:4189: use of closed network connection Dec 28 00:48:30.095583489 (537h19m29.966666524s) [noise=true , address=45.91.116.14:4189 , piece_id=UWTBBQ6Y7UKPQ365HO2JLJHBWYH3MB7EAUSVG22SGL2DTHLY4GNA, seg_pos={0 0}, upload=471240]: Long tail cancellation due to optimal threshold Dec 28 00:48:30.095600162 (537h19m29.966683197s) [seg_pos={0 0}, upload=471240]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:48:30.095663632 (537h19m29.966746667s) [upload=471240]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:48:30.095667961 (537h19m29.966750996s) [upload=471240]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:48:30.385685009 (537h19m30.256768044s) [upload=471240]: Done uploading segments... Dec 28 00:48:30.385703485 (537h19m30.25678652s) [upload=471240]: Upload finished. err: <nil> Duration: 30.003266277s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 16.164799982s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 00:48:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 16.164819219s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 7.793571436s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 00:47:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 7.793589561s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [X....X....X.XXX.X.X.XX.X.XXX...X.X.XX.XXX....X...X.X.X...XXX] duration: 30.001904802s trace: https://jaeger.dev.storj.io/trace/4ce693d9e80d661b FAIL (2025-12-28 00:47:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, Piece ID: AMPWYKZ36CU6M4EUOEQYXHWSYMIMAHANYIB6FEJ7WTG4BLBPPSIQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: YWNMYEMBDYEIBRROGXHS5DGU57ONSIUJPLY6PYVHJPBKTWUHYPTQ) context deadline exceeded ======LOGS====== 28 00:47:21.341440937 (537h18m21.212523972s) [upload=471228]: Starting upload Dec 28 00:47:21.341572969 (537h18m21.212656004s) [upload=471228]: Done starting upload Dec 28 00:47:21.341655516 (537h18m21.212738551s) [upload=471228]: Uploading segments... Dec 28 00:47:21.341878340 (537h18m21.212961375s) [upload=471228]: Got next segment. Inline: false Dec 28 00:47:21.341903468 (537h18m21.212986503s) [upload=471228]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:47:21.341909529 (537h18m21.212992564s) [upload=471228]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:47:21.589680156 (537h18m21.460763201s) [seg_pos={0 0}, upload=471228]: Begin upload segment... Dec 28 00:47:21.589736031 (537h18m21.460819076s) [seg_pos={0 0}, upload=471228]: Done begin upload segment. Dec 28 00:47:21.589774655 (537h18m21.46085769s) [noise=true , address=45.91.116.4:4940 , piece_id=R7RLOUZIV3L5XZK35FFX6RHDF2QN3GINCMDTMWXLKZJNVTKJPMDA, seg_pos={0 0}, upload=471228]: Uploading piece... Dec 28 00:47:21.589784274 (537h18m21.460867319s) [noise=true , address=145.79.227.21:4473 , piece_id=WCIWQ2F3EH2SZEF4BA3SG3PMUBDTWWBYKVO7HYMEQSUQF63R4XJA, seg_pos={0 0}, upload=471228]: Uploading piece... Dec 28 00:47:21.590587894 (537h18m21.461670939s) [noise=true , address=145.79.227.13:4803 , piece_id=2IXEFHNYZ3VW6WT5GN7Z2QQQEIDIYZM5GBK66HUHI3GPVNIRK2EA, seg_pos={0 0}, upload=471228]: Uploading piece... Dec 28 00:47:21.591043802 (537h18m21.462126847s) [noise=true , address=145.79.227.38:4977 , piece_id=ZUNT43KYCGCGQJBHH7JGH2PAZHI4MCIN4XOOTKZU46SBTFWX2WRA, seg_pos={0 0}, upload=471228]: Uploading piece... Dec 28 00:47:21.591437131 (537h18m21.462520166s) [noise=true , address=145.79.227.18:4007 , piece_id=J26VLMOUQM22SO4DENMCR62IX6ZTJJCJHXVCOXFMO2Z2B6WRNVGA, seg_pos={0 0}, upload=471228]: Uploading piece... Dec 28 00:47:21.591805432 (537h18m21.462888477s) [noise=true , address=145.79.227.52:4445 , piece_id=VU4YXCKNMA6QZK2HMJDJMXZPUFJ2JG3RAOC27W7ADOWVSU2BX4RA, seg_pos={0 0}, upload=471228]: Uploading piece... Dec 28 00:47:21.592143246 (537h18m21.463226291s) [noise=true , address=145.79.227.39:4015 , piece_id=FXSDLCTI737SZWNY6LHGOQSF3OH4YU5Q7ECOS2KSL6CJTNVFH7RA, seg_pos={0 0}, upload=471228]: Uploading piece... Dec 28 00:47:21.592419521 (537h18m21.463502556s) [noise=true , address=77.90.31.12:4590 , piece_id=VKG6F7KMQXFYQ77E2JP6XUKZMBW6ZTUCSLSF4QYTVWDCVR2EOP2A, seg_pos={0 0}, upload=471228]: Uploading piece... Dec 28 00:47:21.592687912 (537h18m21.463770948s) [noise=true , address=145.79.227.24:4583 , piece_id=SNC4DALWZATFQCBYBKEVRNG37WIG5NLB3ZCEGXPVKZVT4DBSKGHA, seg_pos={0 0}, upload=471228]: Uploading piece... Dec 28 00:47:21.592955502 (537h18m21.464038537s) [noise=true , address=145.79.227.32:4836 , piece_id=F3XQTUVI53J4PRCCVUKM4235EV4UXNEGBNCCKLGFJHOQFEHILKDQ, seg_pos={0 0}, upload=471228]: Uploading piece... Dec 28 00:47:21.593331008 (537h18m21.464414043s) [noise=true , address=145.79.227.36:4518 , piece_id=TRIPLRL4GDCBHN33MOA2ZDCNG2UVCORRR4UQ6HWPKXE2OTV673OA, seg_pos={0 0}, upload=471228]: Uploading piece... Dec 28 00:47:21.594254235 (537h18m21.46533728s) [noise=true , address=145.79.227.14:4837 , piece_id=75UD6YSDIMODJYMSASKFZL7KFNDDISTRRKTPG5KLDIXB3PIHVCDA, seg_pos={0 0}, upload=471228]: Uploading piece... Dec 28 00:47:21.594805254 (537h18m21.4658883s) [noise=true , address=145.79.227.29:4943 , piece_id=AMPWYKZ36CU6M4EUOEQYXHWSYMIMAHANYIB6FEJ7WTG4BLBPPSIQ, seg_pos={0 0}, upload=471228]: Uploading piece... Dec 28 00:47:21.595253418 (537h18m21.466336453s) [noise=true , address=45.91.116.12:4226 , piece_id=MCUGVFHVIMLHBHZO523CT7ENSB6GK3RAAVY7N76TU4J7IBI53D6Q, seg_pos={0 0}, upload=471228]: Uploading piece... Dec 28 00:47:21.595656244 (537h18m21.4667393s) [noise=true , address=145.79.227.15:4874 , piece_id=WI3BVOHIYUMACAZFWXBNBOSKPXCUVDNFA4374GLAC744CV6QFVGA, seg_pos={0 0}, upload=471228]: Uploading piece... Dec 28 00:47:21.596243693 (537h18m21.467326738s) [noise=true , address=45.91.116.6:4974 , piece_id=YWNMYEMBDYEIBRROGXHS5DGU57ONSIUJPLY6PYVHJPBKTWUHYPTQ, seg_pos={0 0}, upload=471228]: Uploading piece... Dec 28 00:47:23.301091491 (537h18m23.172174536s) [upload=471228]: Next returned nil segment Dec 28 00:47:23.301114685 (537h18m23.17219772s) [upload=471228]: Waiting for error group managing segments... Dec 28 00:47:23.584875633 (537h18m23.455958678s) [noise=true , address=145.79.227.39:4015 , piece_id=FXSDLCTI737SZWNY6LHGOQSF3OH4YU5Q7ECOS2KSL6CJTNVFH7RA, seg_pos={0 0}, upload=471228]: Done uploading piece. err: <nil> Dec 28 00:47:23.594205398 (537h18m23.465288443s) [noise=true , address=145.79.227.52:4445 , piece_id=VU4YXCKNMA6QZK2HMJDJMXZPUFJ2JG3RAOC27W7ADOWVSU2BX4RA, seg_pos={0 0}, upload=471228]: Done uploading piece. err: <nil> Dec 28 00:47:23.598462294 (537h18m23.469545329s) [noise=true , address=145.79.227.36:4518 , piece_id=TRIPLRL4GDCBHN33MOA2ZDCNG2UVCORRR4UQ6HWPKXE2OTV673OA, seg_pos={0 0}, upload=471228]: Done uploading piece. err: <nil> Dec 28 00:47:23.599682567 (537h18m23.470765612s) [noise=true , address=145.79.227.24:4583 , piece_id=SNC4DALWZATFQCBYBKEVRNG37WIG5NLB3ZCEGXPVKZVT4DBSKGHA, seg_pos={0 0}, upload=471228]: Done uploading piece. err: <nil> Dec 28 00:47:23.600141962 (537h18m23.471225007s) [noise=true , address=145.79.227.38:4977 , piece_id=ZUNT43KYCGCGQJBHH7JGH2PAZHI4MCIN4XOOTKZU46SBTFWX2WRA, seg_pos={0 0}, upload=471228]: Done uploading piece. err: <nil> Dec 28 00:47:23.603651765 (537h18m23.47473481s) [noise=true , address=145.79.227.32:4836 , piece_id=F3XQTUVI53J4PRCCVUKM4235EV4UXNEGBNCCKLGFJHOQFEHILKDQ, seg_pos={0 0}, upload=471228]: Done uploading piece. err: <nil> Dec 28 00:47:23.604752631 (537h18m23.475835676s) [noise=true , address=145.79.227.21:4473 , piece_id=WCIWQ2F3EH2SZEF4BA3SG3PMUBDTWWBYKVO7HYMEQSUQF63R4XJA, seg_pos={0 0}, upload=471228]: Done uploading piece. err: <nil> Dec 28 00:47:23.609420950 (537h18m23.480503985s) [noise=true , address=145.79.227.15:4874 , piece_id=WI3BVOHIYUMACAZFWXBNBOSKPXCUVDNFA4374GLAC744CV6QFVGA, seg_pos={0 0}, upload=471228]: Done uploading piece. err: <nil> Dec 28 00:47:23.613529504 (537h18m23.484612549s) [noise=true , address=145.79.227.14:4837 , piece_id=75UD6YSDIMODJYMSASKFZL7KFNDDISTRRKTPG5KLDIXB3PIHVCDA, seg_pos={0 0}, upload=471228]: Done uploading piece. err: <nil> Dec 28 00:47:23.626010911 (537h18m23.497093946s) [noise=true , address=145.79.227.13:4803 , piece_id=2IXEFHNYZ3VW6WT5GN7Z2QQQEIDIYZM5GBK66HUHI3GPVNIRK2EA, seg_pos={0 0}, upload=471228]: Done uploading piece. err: <nil> Dec 28 00:47:23.696589704 (537h18m23.567672739s) [noise=true , address=145.79.227.29:4943 , piece_id=AMPWYKZ36CU6M4EUOEQYXHWSYMIMAHANYIB6FEJ7WTG4BLBPPSIQ, seg_pos={0 0}, upload=471228]: Done uploading piece. err: <nil> Dec 28 00:47:23.828256269 (537h18m23.699339304s) [noise=true , address=45.91.116.6:4974 , piece_id=YWNMYEMBDYEIBRROGXHS5DGU57ONSIUJPLY6PYVHJPBKTWUHYPTQ, seg_pos={0 0}, upload=471228]: Done uploading piece. err: <nil> Dec 28 00:47:23.841676603 (537h18m23.712759638s) [noise=true , address=45.91.116.4:4940 , piece_id=R7RLOUZIV3L5XZK35FFX6RHDF2QN3GINCMDTMWXLKZJNVTKJPMDA, seg_pos={0 0}, upload=471228]: Done uploading piece. err: <nil> Dec 28 00:47:23.841695659 (537h18m23.712778694s) [seg_pos={0 0}, upload=471228]: Segment reached optimal threshold of 13 pieces. Dec 28 00:47:23.841883868 (537h18m23.712966903s) [noise=true , address=77.90.31.12:4590 , piece_id=VKG6F7KMQXFYQ77E2JP6XUKZMBW6ZTUCSLSF4QYTVWDCVR2EOP2A, seg_pos={0 0}, upload=471228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 28 00:47:23.841890270 (537h18m23.712973315s) [noise=true , address=77.90.31.12:4590 , piece_id=VKG6F7KMQXFYQ77E2JP6XUKZMBW6ZTUCSLSF4QYTVWDCVR2EOP2A, seg_pos={0 0}, upload=471228]: Long tail cancellation due to optimal threshold Dec 28 00:47:23.842001111 (537h18m23.713084156s) [noise=true , address=45.91.116.12:4226 , piece_id=MCUGVFHVIMLHBHZO523CT7ENSB6GK3RAAVY7N76TU4J7IBI53D6Q, seg_pos={0 0}, upload=471228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 28 00:47:23.842007382 (537h18m23.713090417s) [noise=true , address=45.91.116.12:4226 , piece_id=MCUGVFHVIMLHBHZO523CT7ENSB6GK3RAAVY7N76TU4J7IBI53D6Q, seg_pos={0 0}, upload=471228]: Long tail cancellation due to optimal threshold Dec 28 00:47:23.842258030 (537h18m23.713341076s) [noise=true , address=145.79.227.18:4007 , piece_id=J26VLMOUQM22SO4DENMCR62IX6ZTJJCJHXVCOXFMO2Z2B6WRNVGA, seg_pos={0 0}, upload=471228]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 188.166.225.170:48114->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:48114->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48114->145.79.227.18:4007: use of closed network connection Dec 28 00:47:23.842288928 (537h18m23.713371974s) [noise=true , address=145.79.227.18:4007 , piece_id=J26VLMOUQM22SO4DENMCR62IX6ZTJJCJHXVCOXFMO2Z2B6WRNVGA, seg_pos={0 0}, upload=471228]: Long tail cancellation due to optimal threshold Dec 28 00:47:23.842323675 (537h18m23.71340671s) [seg_pos={0 0}, upload=471228]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:47:23.842601033 (537h18m23.713684068s) [upload=471228]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:47:23.842609700 (537h18m23.713692745s) [upload=471228]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:47:24.140412746 (537h18m24.011495791s) [upload=471228]: Done uploading segments... Dec 28 00:47:24.140437603 (537h18m24.011520648s) [upload=471228]: Upload finished. err: <nil> Duration: 30.001934298s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 28.470153221s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 00:47:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 28.470180101s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [.X....X....X.XXX.X.X.XX.X.XXX...X.X.XX.XXX....X...X.X.X...XX] duration: 30.002287151s trace: https://jaeger.dev.storj.io/trace/44fbb66e08b3fa39 FAIL (2025-12-28 00:47:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 00:46:44.018287295 (537h17m43.88937034s) [upload=471220]: Starting upload Dec 28 00:46:44.018426681 (537h17m43.889509726s) [upload=471220]: Done starting upload Dec 28 00:46:44.018644867 (537h17m43.889727902s) [upload=471220]: Uploading segments... Dec 28 00:46:44.018785685 (537h17m43.88986873s) [upload=471220]: Got next segment. Inline: false Dec 28 00:46:44.018828126 (537h17m43.889911171s) [upload=471220]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:46:44.018845077 (537h17m43.889928123s) [upload=471220]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:46:44.261538786 (537h17m44.132621831s) [seg_pos={0 0}, upload=471220]: Begin upload segment... Dec 28 00:46:44.261585967 (537h17m44.132669002s) [seg_pos={0 0}, upload=471220]: Done begin upload segment. Dec 28 00:46:44.261613118 (537h17m44.132696153s) [noise=true , address=45.91.116.10:4081 , piece_id=6RXKBJXOF2PH53SDHAQL6GMVH67BGLRIZWYUQ3UXKIK45532FXRQ, seg_pos={0 0}, upload=471220]: Uploading piece... Dec 28 00:46:44.261707899 (537h17m44.132790934s) [noise=true , address=45.91.116.5:4297 , piece_id=B2TDY6RRRADUO2HSN43W7GYA23NPBVNJ7RJHMC2QWQ4MVXK74J3Q, seg_pos={0 0}, upload=471220]: Uploading piece... Dec 28 00:46:44.262532068 (537h17m44.133615103s) [noise=true , address=77.90.31.10:4518 , piece_id=YL4TW45JXMFW3P62T2SHRGZ5GQNODC2BQ4OG3TR24TIPHIANXNNQ, seg_pos={0 0}, upload=471220]: Uploading piece... Dec 28 00:46:44.262920768 (537h17m44.134003813s) [noise=true , address=45.91.116.7:4009 , piece_id=AMQZ5RBDQGAMN2KGJ4LJC7FSLWD46MOCGQ3QCCX4DEX7TZVO75ZA, seg_pos={0 0}, upload=471220]: Uploading piece... Dec 28 00:46:44.263239886 (537h17m44.134322931s) [noise=true , address=45.91.116.4:4937 , piece_id=4QMNEQEVUHEIAF2F6LRZI2C7BXC4NUVSUTYJLDFEGCYQHTUBHVIA, seg_pos={0 0}, upload=471220]: Uploading piece... Dec 28 00:46:44.263508958 (537h17m44.134592003s) [noise=true , address=145.79.227.14:4837 , piece_id=SOLMUAFCXKD3FTX4MXJR7AWJYOM4UQHY3B2VKYIBEWEOT6ILGYXA, seg_pos={0 0}, upload=471220]: Uploading piece... Dec 28 00:46:44.263734818 (537h17m44.134817853s) [noise=true , address=145.79.227.48:4303 , piece_id=BLQLRNCELSAA5SVANNCTHU2I6VCBZZT5WLY3HP6NTTB466637X7Q, seg_pos={0 0}, upload=471220]: Uploading piece... Dec 28 00:46:44.263938796 (537h17m44.135021832s) [noise=true , address=145.79.227.24:4583 , piece_id=W7KGXAFOQOUOFZZODZXPJGT3FVP345RDCPNNPFDEGT2L7X5AQWVQ, seg_pos={0 0}, upload=471220]: Uploading piece... Dec 28 00:46:44.264152183 (537h17m44.135235228s) [noise=true , address=145.79.227.15:4874 , piece_id=AWWKZJYQE7374XJD54IMW7GDJ6NKQX7ASX5JACBWWRLDGKQQMYYQ, seg_pos={0 0}, upload=471220]: Uploading piece... Dec 28 00:46:44.264385728 (537h17m44.135468763s) [noise=true , address=77.90.31.21:4877 , piece_id=DQU4BO3KXWYW644DEUUM47TWF5KNYFVCN6A5UI3BK33X4YF6JIOQ, seg_pos={0 0}, upload=471220]: Uploading piece... Dec 28 00:46:44.264582152 (537h17m44.135665197s) [noise=true , address=77.90.31.17:4773 , piece_id=DO72W4ZKI37EDLVUKSEB6ZEB7AOX4RPONDICCYYDAHBIQM4NCO2Q, seg_pos={0 0}, upload=471220]: Uploading piece... Dec 28 00:46:44.265012461 (537h17m44.136095496s) [noise=true , address=145.79.227.5:4221 , piece_id=77G3DNPXMVVACD6JTITECC65VFY6DXBAMY6GKCFIYJDQ5PYDOC2Q, seg_pos={0 0}, upload=471220]: Uploading piece... Dec 28 00:46:44.265511220 (537h17m44.136594266s) [noise=true , address=145.79.227.32:4836 , piece_id=AIP5PKELXX7DHSD3UDT4UJGMKJP5X5LZONYOBWVAZ2VBYBV22WFA, seg_pos={0 0}, upload=471220]: Uploading piece... Dec 28 00:46:44.265978701 (537h17m44.137061746s) [noise=true , address=145.79.227.18:4006 , piece_id=H5JCKF6M5R7GLGHLTC7AAF2B6M2ZKMU2RC3YEGJA4DWOWMIIRY3A, seg_pos={0 0}, upload=471220]: Uploading piece... Dec 28 00:46:44.266427546 (537h17m44.137510591s) [noise=true , address=145.79.227.3:4151 , piece_id=BY7BER2SPQBK4MD4RKNFXBNZCVU4DUVDVLNB7NFVCBW2UVBUXTDA, seg_pos={0 0}, upload=471220]: Uploading piece... Dec 28 00:46:44.266954690 (537h17m44.138037735s) [noise=true , address=145.79.227.30:4762 , piece_id=QLYVD5VF5O2H5LE22DYR4VN5IJR3SHCYU772X2JQEMZKMZHGNSDQ, seg_pos={0 0}, upload=471220]: Uploading piece... Dec 28 00:46:45.978278819 (537h17m45.849361854s) [upload=471220]: Next returned nil segment Dec 28 00:46:45.978292485 (537h17m45.84937552s) [upload=471220]: Waiting for error group managing segments... Dec 28 00:46:46.264310300 (537h17m46.135393335s) [noise=true , address=145.79.227.14:4837 , piece_id=SOLMUAFCXKD3FTX4MXJR7AWJYOM4UQHY3B2VKYIBEWEOT6ILGYXA, seg_pos={0 0}, upload=471220]: Done uploading piece. err: <nil> Dec 28 00:46:46.267171909 (537h17m46.138254944s) [noise=true , address=145.79.227.15:4874 , piece_id=AWWKZJYQE7374XJD54IMW7GDJ6NKQX7ASX5JACBWWRLDGKQQMYYQ, seg_pos={0 0}, upload=471220]: Done uploading piece. err: <nil> Dec 28 00:46:46.268554412 (537h17m46.139637447s) [noise=true , address=145.79.227.24:4583 , piece_id=W7KGXAFOQOUOFZZODZXPJGT3FVP345RDCPNNPFDEGT2L7X5AQWVQ, seg_pos={0 0}, upload=471220]: Done uploading piece. err: <nil> Dec 28 00:46:46.272951644 (537h17m46.144034679s) [noise=true , address=145.79.227.48:4303 , piece_id=BLQLRNCELSAA5SVANNCTHU2I6VCBZZT5WLY3HP6NTTB466637X7Q, seg_pos={0 0}, upload=471220]: Done uploading piece. err: <nil> Dec 28 00:46:46.299788465 (537h17m46.17087151s) [noise=true , address=145.79.227.32:4836 , piece_id=AIP5PKELXX7DHSD3UDT4UJGMKJP5X5LZONYOBWVAZ2VBYBV22WFA, seg_pos={0 0}, upload=471220]: Done uploading piece. err: <nil> Dec 28 00:46:46.350138761 (537h17m46.221221796s) [noise=true , address=145.79.227.3:4151 , piece_id=BY7BER2SPQBK4MD4RKNFXBNZCVU4DUVDVLNB7NFVCBW2UVBUXTDA, seg_pos={0 0}, upload=471220]: Done uploading piece. err: <nil> Dec 28 00:46:46.454664194 (537h17m46.325747229s) [noise=true , address=77.90.31.17:4773 , piece_id=DO72W4ZKI37EDLVUKSEB6ZEB7AOX4RPONDICCYYDAHBIQM4NCO2Q, seg_pos={0 0}, upload=471220]: Done uploading piece. err: <nil> Dec 28 00:46:46.474014851 (537h17m46.345097896s) [noise=true , address=77.90.31.10:4518 , piece_id=YL4TW45JXMFW3P62T2SHRGZ5GQNODC2BQ4OG3TR24TIPHIANXNNQ, seg_pos={0 0}, upload=471220]: Done uploading piece. err: <nil> Dec 28 00:46:46.491035831 (537h17m46.362118877s) [noise=true , address=45.91.116.5:4297 , piece_id=B2TDY6RRRADUO2HSN43W7GYA23NPBVNJ7RJHMC2QWQ4MVXK74J3Q, seg_pos={0 0}, upload=471220]: Done uploading piece. err: <nil> Dec 28 00:46:46.500221994 (537h17m46.371305039s) [noise=true , address=45.91.116.10:4081 , piece_id=6RXKBJXOF2PH53SDHAQL6GMVH67BGLRIZWYUQ3UXKIK45532FXRQ, seg_pos={0 0}, upload=471220]: Done uploading piece. err: <nil> Dec 28 00:46:46.543359334 (537h17m46.414442379s) [noise=true , address=45.91.116.7:4009 , piece_id=AMQZ5RBDQGAMN2KGJ4LJC7FSLWD46MOCGQ3QCCX4DEX7TZVO75ZA, seg_pos={0 0}, upload=471220]: Done uploading piece. err: <nil> Dec 28 00:46:46.944367419 (537h17m46.815450454s) [noise=true , address=45.91.116.4:4937 , piece_id=4QMNEQEVUHEIAF2F6LRZI2C7BXC4NUVSUTYJLDFEGCYQHTUBHVIA, seg_pos={0 0}, upload=471220]: Done uploading piece. err: <nil> Dec 28 00:46:47.515770904 (537h17m47.386853939s) [noise=true , address=145.79.227.30:4762 , piece_id=QLYVD5VF5O2H5LE22DYR4VN5IJR3SHCYU772X2JQEMZKMZHGNSDQ, seg_pos={0 0}, upload=471220]: Done uploading piece. err: <nil> Dec 28 00:46:47.515792865 (537h17m47.38687591s) [seg_pos={0 0}, upload=471220]: Segment reached optimal threshold of 13 pieces. Dec 28 00:46:47.515999478 (537h17m47.387082513s) [noise=true , address=77.90.31.21:4877 , piece_id=DQU4BO3KXWYW644DEUUM47TWF5KNYFVCN6A5UI3BK33X4YF6JIOQ, seg_pos={0 0}, upload=471220]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 28 00:46:47.516007433 (537h17m47.387090478s) [noise=true , address=77.90.31.21:4877 , piece_id=DQU4BO3KXWYW644DEUUM47TWF5KNYFVCN6A5UI3BK33X4YF6JIOQ, seg_pos={0 0}, upload=471220]: Long tail cancellation due to optimal threshold Dec 28 00:46:47.516066145 (537h17m47.38714918s) [noise=true , address=145.79.227.18:4006 , piece_id=H5JCKF6M5R7GLGHLTC7AAF2B6M2ZKMU2RC3YEGJA4DWOWMIIRY3A, seg_pos={0 0}, upload=471220]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: write tcp 188.166.225.170:39786->145.79.227.18:4006: use of closed network connection; write tcp 188.166.225.170:39786->145.79.227.18:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39786->145.79.227.18:4006: use of closed network connection Dec 28 00:46:47.516076515 (537h17m47.38715956s) [noise=true , address=145.79.227.18:4006 , piece_id=H5JCKF6M5R7GLGHLTC7AAF2B6M2ZKMU2RC3YEGJA4DWOWMIIRY3A, seg_pos={0 0}, upload=471220]: Long tail cancellation due to optimal threshold Dec 28 00:46:47.516091083 (537h17m47.387174128s) [noise=true , address=145.79.227.5:4221 , piece_id=77G3DNPXMVVACD6JTITECC65VFY6DXBAMY6GKCFIYJDQ5PYDOC2Q, seg_pos={0 0}, upload=471220]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Dec 28 00:46:47.516097315 (537h17m47.38718035s) [noise=true , address=145.79.227.5:4221 , piece_id=77G3DNPXMVVACD6JTITECC65VFY6DXBAMY6GKCFIYJDQ5PYDOC2Q, seg_pos={0 0}, upload=471220]: Long tail cancellation due to optimal threshold Dec 28 00:46:47.516113605 (537h17m47.38719664s) [seg_pos={0 0}, upload=471220]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:46:47.516185764 (537h17m47.387268809s) [upload=471220]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:46:47.516191815 (537h17m47.38727485s) [upload=471220]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:46:47.804067446 (537h17m47.675150511s) [upload=471220]: Done uploading segments... Dec 28 00:46:47.804093135 (537h17m47.67517618s) [upload=471220]: Upload finished. err: <nil> Duration: 30.002299264s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 27.583874568s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 00:47:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 27.583906459s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [..X....X....X.XXX.X.X.XX.X.XXX...X.X.XX.XXX....X...X.X.X...X] duration: 30.004269102s trace: https://jaeger.dev.storj.io/trace/728ca010cc85bfe6 FAIL (2025-12-28 00:46:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 28 00:46:13.774946672 (537h17m13.646029717s) [upload=471212]: Starting upload Dec 28 00:46:13.775028317 (537h17m13.646111352s) [upload=471212]: Done starting upload Dec 28 00:46:13.775136372 (537h17m13.646219407s) [upload=471212]: Uploading segments... Dec 28 00:46:13.775374947 (537h17m13.646457982s) [upload=471212]: Got next segment. Inline: false Dec 28 00:46:13.775439710 (537h17m13.646522745s) [upload=471212]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:46:13.775446333 (537h17m13.646529378s) [upload=471212]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:46:14.020274126 (537h17m13.891357171s) [seg_pos={0 0}, upload=471212]: Begin upload segment... Dec 28 00:46:14.020324271 (537h17m13.891407316s) [seg_pos={0 0}, upload=471212]: Done begin upload segment. Dec 28 00:46:14.020384847 (537h17m13.891467892s) [noise=true , address=145.79.227.45:4229 , piece_id=R4VSXQCDK3SMPUTJNUTH2JPC3IDKBTB7U4XWL2HKASHK2DIKN5ZQ, seg_pos={0 0}, upload=471212]: Uploading piece... Dec 28 00:46:14.020425664 (537h17m13.891508699s) [noise=true , address=145.79.227.34:4906 , piece_id=6V3OGABO6KRMS6NPB5P2X4FOMVUKAFY2VAGTEX4WUFUJYS5INKGQ, seg_pos={0 0}, upload=471212]: Uploading piece... Dec 28 00:46:14.020900439 (537h17m13.891983494s) [noise=true , address=145.79.227.50:4374 , piece_id=77JRPADVXS2QCQFH6EQQTMLSWLEX367J5ISY7ZVT3JAJIX2AYQQQ, seg_pos={0 0}, upload=471212]: Uploading piece... Dec 28 00:46:14.021248992 (537h17m13.892332027s) [noise=true , address=145.79.227.42:4123 , piece_id=YG2ACKTY4ZD5J4BCXSCN525WK2RNE5NQIL5M4QZEBGFPYKBT2QSQ, seg_pos={0 0}, upload=471212]: Uploading piece... Dec 28 00:46:14.021526329 (537h17m13.892609374s) [noise=true , address=145.79.227.18:4006 , piece_id=PHOWU3Y3TN7VFQT4SOGGOKGCQIM4XHEBZNVXUMHTM47BKXOGPK2A, seg_pos={0 0}, upload=471212]: Uploading piece... Dec 28 00:46:14.021907285 (537h17m13.89299033s) [noise=true , address=145.79.227.37:4565 , piece_id=UAR5K7LDZHL7HR3S7P5WLFDPNUVBZDYOHKKPGKZUWKGJNPROHZPQ, seg_pos={0 0}, upload=471212]: Uploading piece... Dec 28 00:46:14.022199552 (537h17m13.893282587s) [noise=true , address=77.90.31.12:4590 , piece_id=Y7SXH2TZXTNHWCAFDYWMTNJ6JKHOIT5UWFPZ3O7LCNBNHPXHELQA, seg_pos={0 0}, upload=471212]: Uploading piece... Dec 28 00:46:14.022611456 (537h17m13.893694492s) [noise=true , address=145.79.227.21:4473 , piece_id=K7BSMB7HYSXBGN4VPFSR4ZE3REDE4T7AH6TZMYMOBTKVVCYBGDRA, seg_pos={0 0}, upload=471212]: Uploading piece... Dec 28 00:46:14.023002321 (537h17m13.894085356s) [noise=true , address=77.90.31.14:4664 , piece_id=5S4PA4VPBGXVYR7NHYYI54RFCGAQ2C46KS7AB4P4EM3FOYV4BTFQ, seg_pos={0 0}, upload=471212]: Uploading piece... Dec 28 00:46:14.024180805 (537h17m13.89526384s) [noise=true , address=45.91.116.14:4190 , piece_id=NWYXDMQKGE4W4OYZIFPY6JELLIVRFHWII65FRDRJZXSL47UPTACQ, seg_pos={0 0}, upload=471212]: Uploading piece... Dec 28 00:46:14.024316333 (537h17m13.895399368s) [noise=true , address=145.79.227.17:4970 , piece_id=6HMPYEJMCU2H5CWMNYJE3TD6MGTTBVE4DZ2N5PDTDPXX3VFJ4XXA, seg_pos={0 0}, upload=471212]: Uploading piece... Dec 28 00:46:14.025531256 (537h17m13.896614291s) [noise=true , address=145.79.227.15:4876 , piece_id=H7HGMVKOZWDVALPQYWMI5NMQNHMFINGJKWRJCPJOQP5FG26RVPSQ, seg_pos={0 0}, upload=471212]: Uploading piece... Dec 28 00:46:14.025872716 (537h17m13.896955751s) [noise=true , address=45.91.116.4:4939 , piece_id=X52MPGUMVEHIW5TZPQDTZ3M7HDQ6RVG4LFLAM6P7WMTVWC6BZAGA, seg_pos={0 0}, upload=471212]: Uploading piece... Dec 28 00:46:14.027073894 (537h17m13.898156939s) [noise=true , address=145.79.227.36:4518 , piece_id=HJ7B4PUJARXZ2AKXMDMDTDS2X2KIM7GQWAOHJPLBKYP4XML7LAZQ, seg_pos={0 0}, upload=471212]: Uploading piece... Dec 28 00:46:14.028601903 (537h17m13.899684938s) [noise=true , address=145.79.227.38:4978 , piece_id=K234VL6I4GKZ2EX5V3NM5J4SNLJ7YNY7O3JVEGWBQ4TZUJSRBOHQ, seg_pos={0 0}, upload=471212]: Uploading piece... Dec 28 00:46:14.029862554 (537h17m13.900945599s) [noise=true , address=45.91.116.11:4117 , piece_id=EL3FM5XKPWR3J7GYOZP66ZKF4YHTEA3FRJTRVH3KW3KM42FU7S5Q, seg_pos={0 0}, upload=471212]: Uploading piece... Dec 28 00:46:15.738319313 (537h17m15.609402358s) [upload=471212]: Next returned nil segment Dec 28 00:46:15.738335423 (537h17m15.609418468s) [upload=471212]: Waiting for error group managing segments... Dec 28 00:46:16.022097252 (537h17m15.893180297s) [noise=true , address=145.79.227.45:4229 , piece_id=R4VSXQCDK3SMPUTJNUTH2JPC3IDKBTB7U4XWL2HKASHK2DIKN5ZQ, seg_pos={0 0}, upload=471212]: Done uploading piece. err: <nil> Dec 28 00:46:16.024211578 (537h17m15.895294613s) [noise=true , address=145.79.227.15:4876 , piece_id=H7HGMVKOZWDVALPQYWMI5NMQNHMFINGJKWRJCPJOQP5FG26RVPSQ, seg_pos={0 0}, upload=471212]: Done uploading piece. err: <nil> Dec 28 00:46:16.035687369 (537h17m15.906770404s) [noise=true , address=145.79.227.37:4565 , piece_id=UAR5K7LDZHL7HR3S7P5WLFDPNUVBZDYOHKKPGKZUWKGJNPROHZPQ, seg_pos={0 0}, upload=471212]: Done uploading piece. err: <nil> Dec 28 00:46:16.043597301 (537h17m15.914680346s) [noise=true , address=145.79.227.50:4374 , piece_id=77JRPADVXS2QCQFH6EQQTMLSWLEX367J5ISY7ZVT3JAJIX2AYQQQ, seg_pos={0 0}, upload=471212]: Done uploading piece. err: <nil> Dec 28 00:46:16.043642086 (537h17m15.914725131s) [noise=true , address=145.79.227.17:4970 , piece_id=6HMPYEJMCU2H5CWMNYJE3TD6MGTTBVE4DZ2N5PDTDPXX3VFJ4XXA, seg_pos={0 0}, upload=471212]: Done uploading piece. err: <nil> Dec 28 00:46:16.044058930 (537h17m15.915141965s) [noise=true , address=145.79.227.38:4978 , piece_id=K234VL6I4GKZ2EX5V3NM5J4SNLJ7YNY7O3JVEGWBQ4TZUJSRBOHQ, seg_pos={0 0}, upload=471212]: Done uploading piece. err: <nil> Dec 28 00:46:16.044086883 (537h17m15.915169918s) [noise=true , address=145.79.227.21:4473 , piece_id=K7BSMB7HYSXBGN4VPFSR4ZE3REDE4T7AH6TZMYMOBTKVVCYBGDRA, seg_pos={0 0}, upload=471212]: Done uploading piece. err: <nil> Dec 28 00:46:16.056132590 (537h17m15.927215625s) [noise=true , address=145.79.227.42:4123 , piece_id=YG2ACKTY4ZD5J4BCXSCN525WK2RNE5NQIL5M4QZEBGFPYKBT2QSQ, seg_pos={0 0}, upload=471212]: Done uploading piece. err: <nil> Dec 28 00:46:16.061313065 (537h17m15.9323961s) [noise=true , address=145.79.227.36:4518 , piece_id=HJ7B4PUJARXZ2AKXMDMDTDS2X2KIM7GQWAOHJPLBKYP4XML7LAZQ, seg_pos={0 0}, upload=471212]: Done uploading piece. err: <nil> Dec 28 00:46:16.288534041 (537h17m16.159617076s) [noise=true , address=77.90.31.14:4664 , piece_id=5S4PA4VPBGXVYR7NHYYI54RFCGAQ2C46KS7AB4P4EM3FOYV4BTFQ, seg_pos={0 0}, upload=471212]: Done uploading piece. err: <nil> Dec 28 00:46:16.526942569 (537h17m16.398025604s) [noise=true , address=77.90.31.12:4590 , piece_id=Y7SXH2TZXTNHWCAFDYWMTNJ6JKHOIT5UWFPZ3O7LCNBNHPXHELQA, seg_pos={0 0}, upload=471212]: Done uploading piece. err: <nil> Dec 28 00:46:16.983879912 (537h17m16.854962947s) [noise=true , address=45.91.116.14:4190 , piece_id=NWYXDMQKGE4W4OYZIFPY6JELLIVRFHWII65FRDRJZXSL47UPTACQ, seg_pos={0 0}, upload=471212]: Done uploading piece. err: <nil> Dec 28 00:46:17.568773614 (537h17m17.439856659s) [noise=true , address=45.91.116.4:4939 , piece_id=X52MPGUMVEHIW5TZPQDTZ3M7HDQ6RVG4LFLAM6P7WMTVWC6BZAGA, seg_pos={0 0}, upload=471212]: Done uploading piece. err: <nil> Dec 28 00:46:17.568798612 (537h17m17.439881657s) [seg_pos={0 0}, upload=471212]: Segment reached optimal threshold of 13 pieces. Dec 28 00:46:17.569009323 (537h17m17.440092358s) [noise=true , address=145.79.227.18:4006 , piece_id=PHOWU3Y3TN7VFQT4SOGGOKGCQIM4XHEBZNVXUMHTM47BKXOGPK2A, seg_pos={0 0}, upload=471212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc): protocol: write tcp 188.166.225.170:59800->145.79.227.18:4006: use of closed network connection; write tcp 188.166.225.170:59800->145.79.227.18:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59800->145.79.227.18:4006: use of closed network connection Dec 28 00:46:17.569018209 (537h17m17.440101265s) [noise=true , address=145.79.227.18:4006 , piece_id=PHOWU3Y3TN7VFQT4SOGGOKGCQIM4XHEBZNVXUMHTM47BKXOGPK2A, seg_pos={0 0}, upload=471212]: Long tail cancellation due to optimal threshold Dec 28 00:46:17.569430575 (537h17m17.44051363s) [noise=true , address=45.91.116.11:4117 , piece_id=EL3FM5XKPWR3J7GYOZP66ZKF4YHTEA3FRJTRVH3KW3KM42FU7S5Q, seg_pos={0 0}, upload=471212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 28 00:46:17.569451565 (537h17m17.44053461s) [noise=true , address=45.91.116.11:4117 , piece_id=EL3FM5XKPWR3J7GYOZP66ZKF4YHTEA3FRJTRVH3KW3KM42FU7S5Q, seg_pos={0 0}, upload=471212]: Long tail cancellation due to optimal threshold Dec 28 00:46:17.569462536 (537h17m17.440545581s) [noise=true , address=145.79.227.34:4906 , piece_id=6V3OGABO6KRMS6NPB5P2X4FOMVUKAFY2VAGTEX4WUFUJYS5INKGQ, seg_pos={0 0}, upload=471212]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 28 00:46:17.569472805 (537h17m17.44055585s) [noise=true , address=145.79.227.34:4906 , piece_id=6V3OGABO6KRMS6NPB5P2X4FOMVUKAFY2VAGTEX4WUFUJYS5INKGQ, seg_pos={0 0}, upload=471212]: Long tail cancellation due to optimal threshold Dec 28 00:46:17.569494767 (537h17m17.440577802s) [seg_pos={0 0}, upload=471212]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:46:17.569585479 (537h17m17.440668514s) [upload=471212]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:46:17.569591341 (537h17m17.440674386s) [upload=471212]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:46:17.895112075 (537h17m17.76619512s) [upload=471212]: Done uploading segments... Dec 28 00:46:17.895132875 (537h17m17.76621591s) [upload=471212]: Upload finished. err: <nil> Duration: 30.004296575s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 17.79382507s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 00:46:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 17.793845308s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 32.471947697s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 00:45:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 32.471982442s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [...X....X....X.XXX.X.X.XX.X.XXX...X.X.XX.XXX....X...X.X.X...] duration: 5.923734576s trace: https://jaeger.dev.storj.io/trace/4b7e763d46d698c6 FAIL (2025-12-28 00:45:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: PUZ3ZS4DJK2OPM225MBMYTSGTY5LKSR74BQJET5YAUY6HDLTAS4A) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: 2EQZNKY2ZNWUGDB2TINXJEZTSPGVDDWHRJAHPKWNW3B7R6FRZHLA) context deadline exceeded ======LOGS====== Dec 28 00:43:13.657501310 (537h14m13.528584356s) [upload=471165]: Starting upload Dec 28 00:43:13.657618114 (537h14m13.528701149s) [upload=471165]: Done starting upload Dec 28 00:43:13.657904789 (537h14m13.528987834s) [upload=471165]: Uploading segments... Dec 28 00:43:13.657954253 (537h14m13.529037298s) [upload=471165]: Got next segment. Inline: false Dec 28 00:43:13.657987216 (537h14m13.529070261s) [upload=471165]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:43:13.657992906 (537h14m13.529075951s) [upload=471165]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:43:13.900922884 (537h14m13.772005919s) [seg_pos={0 0}, upload=471165]: Begin upload segment... Dec 28 00:43:13.900971277 (537h14m13.772054312s) [seg_pos={0 0}, upload=471165]: Done begin upload segment. Dec 28 00:43:13.901019599 (537h14m13.772102634s) [noise=true , address=145.79.227.46:4603 , piece_id=G32YOVG6LJIDJMJETLHMTFGRSLRWIEUCIYPWUF6G27WSYOFQURMA, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.901041701 (537h14m13.772124736s) [noise=true , address=145.79.227.24:4583 , piece_id=ALFCBGFFGMCJIU6ULD36CMUKVI37W67UEWFS3HLSNCYHCXY6AJ7A, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.901578393 (537h14m13.772661438s) [noise=true , address=45.91.116.11:4120 , piece_id=MAI5L5BM3XII7CSOIIWSXUNNSBYHWNOUWHWP3HK6S3Z4LAAMSHLQ, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.902086200 (537h14m13.773169245s) [noise=true , address=45.91.116.12:4225 , piece_id=W2YMTTDYUV2WRCBAZG2U45FI3ISHOV5BLZDKNTLWNFFUYLIHCONA, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.902229984 (537h14m13.773313019s) [noise=true , address=77.90.31.13:4629 , piece_id=HO4IKUBX4OVQI2OYEPVKZTPGIRWEQQ37PIYC2UDBTGSKTLQUOHKQ, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.903164243 (537h14m13.774247278s) [noise=true , address=77.90.31.11:4556 , piece_id=2EQZNKY2ZNWUGDB2TINXJEZTSPGVDDWHRJAHPKWNW3B7R6FRZHLA, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.903549527 (537h14m13.774632572s) [noise=true , address=145.79.227.19:4077 , piece_id=DEEZHS5GIOTHRTCLQTFHNCR2SGIV3TFCB2BH2G2HVJ7ELRO3SDZA, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.903713268 (537h14m13.774796303s) [noise=true , address=145.79.227.41:4087 , piece_id=U6JCLCWNJDVHFOWWD3RTMOG5XYGUWXJORTV2BLKMF55B6L23FHMA, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.903887008 (537h14m13.774970044s) [noise=true , address=145.79.227.17:4971 , piece_id=UFXPZK7CFMXD7PSENRMMGI34U7JZDJFGSYZT7ZUJCESTHCHBIJBQ, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.904303001 (537h14m13.775386036s) [noise=true , address=145.79.227.3:4151 , piece_id=WISCT4SLYWN63FJ7OXB5HO7JOVJXQVDS6KICAQE3NQFBSVEJFKPQ, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.904620756 (537h14m13.775703821s) [noise=true , address=145.79.227.4:4185 , piece_id=UA5F6MOZG2L537H3DIJCRKJAEOQCGI2BMQGL4A7SLWTLFEAOMSUA, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.904877245 (537h14m13.77596028s) [noise=true , address=45.91.116.7:4010 , piece_id=PINGAXWR4WELATNHIQI7BNEGK4YZH7SEQZ6OBRI7XTEB2XMHAX3A, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.905002543 (537h14m13.776085578s) [noise=true , address=145.79.227.48:4301 , piece_id=DEONEQZHWS2RGAYCNK34WAH7TQXUN6CPXOLYGJDA6CEOFDEH4LLQ, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.905241498 (537h14m13.776324573s) [noise=true , address=45.91.116.13:4155 , piece_id=QBZZDRSWXDDOPRD2ZNO54S2UA7LRJZBWATIB57O5ETH4F2M7WQMA, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.905606823 (537h14m13.776689858s) [noise=true , address=45.91.116.8:4046 , piece_id=HTSRGPOSCX2WFOGS4GXJTPUTIRVZCQJUN4CG4QKLK3SJUC6AZLQA, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.905950037 (537h14m13.777033082s) [noise=true , address=77.90.31.15:4697 , piece_id=PUZ3ZS4DJK2OPM225MBMYTSGTY5LKSR74BQJET5YAUY6HDLTAS4A, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:15.623274913 (537h14m15.494357969s) [upload=471165]: Next returned nil segment Dec 28 00:43:15.623296375 (537h14m15.49437941s) [upload=471165]: Waiting for error group managing segments... Dec 28 00:43:15.907940263 (537h14m15.779023308s) [noise=true , address=145.79.227.17:4971 , piece_id=UFXPZK7CFMXD7PSENRMMGI34U7JZDJFGSYZT7ZUJCESTHCHBIJBQ, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:15.911507425 (537h14m15.78259047s) [noise=true , address=145.79.227.24:4583 , piece_id=ALFCBGFFGMCJIU6ULD36CMUKVI37W67UEWFS3HLSNCYHCXY6AJ7A, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:15.913055183 (537h14m15.784138258s) [noise=true , address=145.79.227.3:4151 , piece_id=WISCT4SLYWN63FJ7OXB5HO7JOVJXQVDS6KICAQE3NQFBSVEJFKPQ, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:15.931205985 (537h14m15.80228903s) [noise=true , address=145.79.227.41:4087 , piece_id=U6JCLCWNJDVHFOWWD3RTMOG5XYGUWXJORTV2BLKMF55B6L23FHMA, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:15.931599654 (537h14m15.802682689s) [noise=true , address=145.79.227.19:4077 , piece_id=DEEZHS5GIOTHRTCLQTFHNCR2SGIV3TFCB2BH2G2HVJ7ELRO3SDZA, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:15.936567945 (537h14m15.80765099s) [noise=true , address=145.79.227.48:4301 , piece_id=DEONEQZHWS2RGAYCNK34WAH7TQXUN6CPXOLYGJDA6CEOFDEH4LLQ, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:15.939257806 (537h14m15.810340841s) [noise=true , address=145.79.227.4:4185 , piece_id=UA5F6MOZG2L537H3DIJCRKJAEOQCGI2BMQGL4A7SLWTLFEAOMSUA, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:16.082116904 (537h14m15.953199939s) [noise=true , address=77.90.31.15:4697 , piece_id=PUZ3ZS4DJK2OPM225MBMYTSGTY5LKSR74BQJET5YAUY6HDLTAS4A, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:16.150063445 (537h14m16.02114649s) [noise=true , address=45.91.116.13:4155 , piece_id=QBZZDRSWXDDOPRD2ZNO54S2UA7LRJZBWATIB57O5ETH4F2M7WQMA, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:16.169105583 (537h14m16.040188628s) [noise=true , address=77.90.31.13:4629 , piece_id=HO4IKUBX4OVQI2OYEPVKZTPGIRWEQQ37PIYC2UDBTGSKTLQUOHKQ, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:16.194315498 (537h14m16.065398543s) [noise=true , address=45.91.116.7:4010 , piece_id=PINGAXWR4WELATNHIQI7BNEGK4YZH7SEQZ6OBRI7XTEB2XMHAX3A, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:16.220780552 (537h14m16.091863587s) [noise=true , address=45.91.116.8:4046 , piece_id=HTSRGPOSCX2WFOGS4GXJTPUTIRVZCQJUN4CG4QKLK3SJUC6AZLQA, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:16.839589454 (537h14m16.710672499s) [noise=true , address=77.90.31.11:4556 , piece_id=2EQZNKY2ZNWUGDB2TINXJEZTSPGVDDWHRJAHPKWNW3B7R6FRZHLA, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:16.839615473 (537h14m16.710698518s) [seg_pos={0 0}, upload=471165]: Segment reached optimal threshold of 13 pieces. Dec 28 00:43:16.839833949 (537h14m16.710916994s) [noise=true , address=45.91.116.11:4120 , piece_id=MAI5L5BM3XII7CSOIIWSXUNNSBYHWNOUWHWP3HK6S3Z4LAAMSHLQ, seg_pos={0 0}, upload=471165]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 28 00:43:16.839842255 (537h14m16.7109253s) [noise=true , address=45.91.116.11:4120 , piece_id=MAI5L5BM3XII7CSOIIWSXUNNSBYHWNOUWHWP3HK6S3Z4LAAMSHLQ, seg_pos={0 0}, upload=471165]: Long tail cancellation due to optimal threshold Dec 28 00:43:16.839971551 (537h14m16.711054586s) [noise=true , address=145.79.227.46:4603 , piece_id=G32YOVG6LJIDJMJETLHMTFGRSLRWIEUCIYPWUF6G27WSYOFQURMA, seg_pos={0 0}, upload=471165]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX): protocol: expected piece hash; context canceled; EOF Dec 28 00:43:16.840002039 (537h14m16.711085084s) [noise=true , address=145.79.227.46:4603 , piece_id=G32YOVG6LJIDJMJETLHMTFGRSLRWIEUCIYPWUF6G27WSYOFQURMA, seg_pos={0 0}, upload=471165]: Long tail cancellation due to optimal threshold Dec 28 00:43:16.840160532 (537h14m16.711243577s) [noise=true , address=45.91.116.12:4225 , piece_id=W2YMTTDYUV2WRCBAZG2U45FI3ISHOV5BLZDKNTLWNFFUYLIHCONA, seg_pos={0 0}, upload=471165]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 28 00:43:16.840187933 (537h14m16.711270968s) [noise=true , address=45.91.116.12:4225 , piece_id=W2YMTTDYUV2WRCBAZG2U45FI3ISHOV5BLZDKNTLWNFFUYLIHCONA, seg_pos={0 0}, upload=471165]: Long tail cancellation due to optimal threshold Dec 28 00:43:16.840206869 (537h14m16.711289915s) [seg_pos={0 0}, upload=471165]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:43:16.840440475 (537h14m16.71152352s) [upload=471165]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:43:16.840446817 (537h14m16.711529852s) [upload=471165]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:43:17.132569025 (537h14m17.00365206s) [upload=471165]: Done uploading segments... Dec 28 00:43:17.132587990 (537h14m17.003671036s) [upload=471165]: Upload finished. err: <nil> Duration: 5.923758189s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 29.754437241s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 00:45:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 29.754467118s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [X...X....X....X.XXX.X.X.XX.X.XXX...X.X.XX.XXX....X...X.X.X..] duration: 8.155755682s trace: https://jaeger.dev.storj.io/trace/4b7e763d46d698c6 FAIL (2025-12-28 00:44:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: PUZ3ZS4DJK2OPM225MBMYTSGTY5LKSR74BQJET5YAUY6HDLTAS4A) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: 2EQZNKY2ZNWUGDB2TINXJEZTSPGVDDWHRJAHPKWNW3B7R6FRZHLA) context deadline exceeded ======LOGS====== Dec 28 00:43:13.657501310 (537h14m13.528584356s) [upload=471165]: Starting upload Dec 28 00:43:13.657618114 (537h14m13.528701149s) [upload=471165]: Done starting upload Dec 28 00:43:13.657904789 (537h14m13.528987834s) [upload=471165]: Uploading segments... Dec 28 00:43:13.657954253 (537h14m13.529037298s) [upload=471165]: Got next segment. Inline: false Dec 28 00:43:13.657987216 (537h14m13.529070261s) [upload=471165]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:43:13.657992906 (537h14m13.529075951s) [upload=471165]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:43:13.900922884 (537h14m13.772005919s) [seg_pos={0 0}, upload=471165]: Begin upload segment... Dec 28 00:43:13.900971277 (537h14m13.772054312s) [seg_pos={0 0}, upload=471165]: Done begin upload segment. Dec 28 00:43:13.901019599 (537h14m13.772102634s) [noise=true , address=145.79.227.46:4603 , piece_id=G32YOVG6LJIDJMJETLHMTFGRSLRWIEUCIYPWUF6G27WSYOFQURMA, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.901041701 (537h14m13.772124736s) [noise=true , address=145.79.227.24:4583 , piece_id=ALFCBGFFGMCJIU6ULD36CMUKVI37W67UEWFS3HLSNCYHCXY6AJ7A, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.901578393 (537h14m13.772661438s) [noise=true , address=45.91.116.11:4120 , piece_id=MAI5L5BM3XII7CSOIIWSXUNNSBYHWNOUWHWP3HK6S3Z4LAAMSHLQ, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.902086200 (537h14m13.773169245s) [noise=true , address=45.91.116.12:4225 , piece_id=W2YMTTDYUV2WRCBAZG2U45FI3ISHOV5BLZDKNTLWNFFUYLIHCONA, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.902229984 (537h14m13.773313019s) [noise=true , address=77.90.31.13:4629 , piece_id=HO4IKUBX4OVQI2OYEPVKZTPGIRWEQQ37PIYC2UDBTGSKTLQUOHKQ, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.903164243 (537h14m13.774247278s) [noise=true , address=77.90.31.11:4556 , piece_id=2EQZNKY2ZNWUGDB2TINXJEZTSPGVDDWHRJAHPKWNW3B7R6FRZHLA, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.903549527 (537h14m13.774632572s) [noise=true , address=145.79.227.19:4077 , piece_id=DEEZHS5GIOTHRTCLQTFHNCR2SGIV3TFCB2BH2G2HVJ7ELRO3SDZA, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.903713268 (537h14m13.774796303s) [noise=true , address=145.79.227.41:4087 , piece_id=U6JCLCWNJDVHFOWWD3RTMOG5XYGUWXJORTV2BLKMF55B6L23FHMA, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.903887008 (537h14m13.774970044s) [noise=true , address=145.79.227.17:4971 , piece_id=UFXPZK7CFMXD7PSENRMMGI34U7JZDJFGSYZT7ZUJCESTHCHBIJBQ, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.904303001 (537h14m13.775386036s) [noise=true , address=145.79.227.3:4151 , piece_id=WISCT4SLYWN63FJ7OXB5HO7JOVJXQVDS6KICAQE3NQFBSVEJFKPQ, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.904620756 (537h14m13.775703821s) [noise=true , address=145.79.227.4:4185 , piece_id=UA5F6MOZG2L537H3DIJCRKJAEOQCGI2BMQGL4A7SLWTLFEAOMSUA, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.904877245 (537h14m13.77596028s) [noise=true , address=45.91.116.7:4010 , piece_id=PINGAXWR4WELATNHIQI7BNEGK4YZH7SEQZ6OBRI7XTEB2XMHAX3A, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.905002543 (537h14m13.776085578s) [noise=true , address=145.79.227.48:4301 , piece_id=DEONEQZHWS2RGAYCNK34WAH7TQXUN6CPXOLYGJDA6CEOFDEH4LLQ, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.905241498 (537h14m13.776324573s) [noise=true , address=45.91.116.13:4155 , piece_id=QBZZDRSWXDDOPRD2ZNO54S2UA7LRJZBWATIB57O5ETH4F2M7WQMA, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.905606823 (537h14m13.776689858s) [noise=true , address=45.91.116.8:4046 , piece_id=HTSRGPOSCX2WFOGS4GXJTPUTIRVZCQJUN4CG4QKLK3SJUC6AZLQA, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.905950037 (537h14m13.777033082s) [noise=true , address=77.90.31.15:4697 , piece_id=PUZ3ZS4DJK2OPM225MBMYTSGTY5LKSR74BQJET5YAUY6HDLTAS4A, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:15.623274913 (537h14m15.494357969s) [upload=471165]: Next returned nil segment Dec 28 00:43:15.623296375 (537h14m15.49437941s) [upload=471165]: Waiting for error group managing segments... Dec 28 00:43:15.907940263 (537h14m15.779023308s) [noise=true , address=145.79.227.17:4971 , piece_id=UFXPZK7CFMXD7PSENRMMGI34U7JZDJFGSYZT7ZUJCESTHCHBIJBQ, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:15.911507425 (537h14m15.78259047s) [noise=true , address=145.79.227.24:4583 , piece_id=ALFCBGFFGMCJIU6ULD36CMUKVI37W67UEWFS3HLSNCYHCXY6AJ7A, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:15.913055183 (537h14m15.784138258s) [noise=true , address=145.79.227.3:4151 , piece_id=WISCT4SLYWN63FJ7OXB5HO7JOVJXQVDS6KICAQE3NQFBSVEJFKPQ, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:15.931205985 (537h14m15.80228903s) [noise=true , address=145.79.227.41:4087 , piece_id=U6JCLCWNJDVHFOWWD3RTMOG5XYGUWXJORTV2BLKMF55B6L23FHMA, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:15.931599654 (537h14m15.802682689s) [noise=true , address=145.79.227.19:4077 , piece_id=DEEZHS5GIOTHRTCLQTFHNCR2SGIV3TFCB2BH2G2HVJ7ELRO3SDZA, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:15.936567945 (537h14m15.80765099s) [noise=true , address=145.79.227.48:4301 , piece_id=DEONEQZHWS2RGAYCNK34WAH7TQXUN6CPXOLYGJDA6CEOFDEH4LLQ, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:15.939257806 (537h14m15.810340841s) [noise=true , address=145.79.227.4:4185 , piece_id=UA5F6MOZG2L537H3DIJCRKJAEOQCGI2BMQGL4A7SLWTLFEAOMSUA, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:16.082116904 (537h14m15.953199939s) [noise=true , address=77.90.31.15:4697 , piece_id=PUZ3ZS4DJK2OPM225MBMYTSGTY5LKSR74BQJET5YAUY6HDLTAS4A, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:16.150063445 (537h14m16.02114649s) [noise=true , address=45.91.116.13:4155 , piece_id=QBZZDRSWXDDOPRD2ZNO54S2UA7LRJZBWATIB57O5ETH4F2M7WQMA, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:16.169105583 (537h14m16.040188628s) [noise=true , address=77.90.31.13:4629 , piece_id=HO4IKUBX4OVQI2OYEPVKZTPGIRWEQQ37PIYC2UDBTGSKTLQUOHKQ, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:16.194315498 (537h14m16.065398543s) [noise=true , address=45.91.116.7:4010 , piece_id=PINGAXWR4WELATNHIQI7BNEGK4YZH7SEQZ6OBRI7XTEB2XMHAX3A, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:16.220780552 (537h14m16.091863587s) [noise=true , address=45.91.116.8:4046 , piece_id=HTSRGPOSCX2WFOGS4GXJTPUTIRVZCQJUN4CG4QKLK3SJUC6AZLQA, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:16.839589454 (537h14m16.710672499s) [noise=true , address=77.90.31.11:4556 , piece_id=2EQZNKY2ZNWUGDB2TINXJEZTSPGVDDWHRJAHPKWNW3B7R6FRZHLA, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:16.839615473 (537h14m16.710698518s) [seg_pos={0 0}, upload=471165]: Segment reached optimal threshold of 13 pieces. Dec 28 00:43:16.839833949 (537h14m16.710916994s) [noise=true , address=45.91.116.11:4120 , piece_id=MAI5L5BM3XII7CSOIIWSXUNNSBYHWNOUWHWP3HK6S3Z4LAAMSHLQ, seg_pos={0 0}, upload=471165]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 28 00:43:16.839842255 (537h14m16.7109253s) [noise=true , address=45.91.116.11:4120 , piece_id=MAI5L5BM3XII7CSOIIWSXUNNSBYHWNOUWHWP3HK6S3Z4LAAMSHLQ, seg_pos={0 0}, upload=471165]: Long tail cancellation due to optimal threshold Dec 28 00:43:16.839971551 (537h14m16.711054586s) [noise=true , address=145.79.227.46:4603 , piece_id=G32YOVG6LJIDJMJETLHMTFGRSLRWIEUCIYPWUF6G27WSYOFQURMA, seg_pos={0 0}, upload=471165]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX): protocol: expected piece hash; context canceled; EOF Dec 28 00:43:16.840002039 (537h14m16.711085084s) [noise=true , address=145.79.227.46:4603 , piece_id=G32YOVG6LJIDJMJETLHMTFGRSLRWIEUCIYPWUF6G27WSYOFQURMA, seg_pos={0 0}, upload=471165]: Long tail cancellation due to optimal threshold Dec 28 00:43:16.840160532 (537h14m16.711243577s) [noise=true , address=45.91.116.12:4225 , piece_id=W2YMTTDYUV2WRCBAZG2U45FI3ISHOV5BLZDKNTLWNFFUYLIHCONA, seg_pos={0 0}, upload=471165]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 28 00:43:16.840187933 (537h14m16.711270968s) [noise=true , address=45.91.116.12:4225 , piece_id=W2YMTTDYUV2WRCBAZG2U45FI3ISHOV5BLZDKNTLWNFFUYLIHCONA, seg_pos={0 0}, upload=471165]: Long tail cancellation due to optimal threshold Dec 28 00:43:16.840206869 (537h14m16.711289915s) [seg_pos={0 0}, upload=471165]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:43:16.840440475 (537h14m16.71152352s) [upload=471165]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:43:16.840446817 (537h14m16.711529852s) [upload=471165]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:43:17.132569025 (537h14m17.00365206s) [upload=471165]: Done uploading segments... Dec 28 00:43:17.132587990 (537h14m17.003671036s) [upload=471165]: Upload finished. err: <nil> Duration: 8.155779967s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 35.239733642s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 00:44:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 35.239766385s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.X...X....X....X.XXX.X.X.XX.X.XXX...X.X.XX.XXX....X...X.X.X.] duration: 8.695471875s trace: https://jaeger.dev.storj.io/trace/4b7e763d46d698c6 FAIL (2025-12-28 00:43:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: PUZ3ZS4DJK2OPM225MBMYTSGTY5LKSR74BQJET5YAUY6HDLTAS4A) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: 2EQZNKY2ZNWUGDB2TINXJEZTSPGVDDWHRJAHPKWNW3B7R6FRZHLA) context deadline exceeded ======LOGS====== Dec 28 00:43:13.657501310 (537h14m13.528584356s) [upload=471165]: Starting upload Dec 28 00:43:13.657618114 (537h14m13.528701149s) [upload=471165]: Done starting upload Dec 28 00:43:13.657904789 (537h14m13.528987834s) [upload=471165]: Uploading segments... Dec 28 00:43:13.657954253 (537h14m13.529037298s) [upload=471165]: Got next segment. Inline: false Dec 28 00:43:13.657987216 (537h14m13.529070261s) [upload=471165]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:43:13.657992906 (537h14m13.529075951s) [upload=471165]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:43:13.900922884 (537h14m13.772005919s) [seg_pos={0 0}, upload=471165]: Begin upload segment... Dec 28 00:43:13.900971277 (537h14m13.772054312s) [seg_pos={0 0}, upload=471165]: Done begin upload segment. Dec 28 00:43:13.901019599 (537h14m13.772102634s) [noise=true , address=145.79.227.46:4603 , piece_id=G32YOVG6LJIDJMJETLHMTFGRSLRWIEUCIYPWUF6G27WSYOFQURMA, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.901041701 (537h14m13.772124736s) [noise=true , address=145.79.227.24:4583 , piece_id=ALFCBGFFGMCJIU6ULD36CMUKVI37W67UEWFS3HLSNCYHCXY6AJ7A, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.901578393 (537h14m13.772661438s) [noise=true , address=45.91.116.11:4120 , piece_id=MAI5L5BM3XII7CSOIIWSXUNNSBYHWNOUWHWP3HK6S3Z4LAAMSHLQ, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.902086200 (537h14m13.773169245s) [noise=true , address=45.91.116.12:4225 , piece_id=W2YMTTDYUV2WRCBAZG2U45FI3ISHOV5BLZDKNTLWNFFUYLIHCONA, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.902229984 (537h14m13.773313019s) [noise=true , address=77.90.31.13:4629 , piece_id=HO4IKUBX4OVQI2OYEPVKZTPGIRWEQQ37PIYC2UDBTGSKTLQUOHKQ, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.903164243 (537h14m13.774247278s) [noise=true , address=77.90.31.11:4556 , piece_id=2EQZNKY2ZNWUGDB2TINXJEZTSPGVDDWHRJAHPKWNW3B7R6FRZHLA, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.903549527 (537h14m13.774632572s) [noise=true , address=145.79.227.19:4077 , piece_id=DEEZHS5GIOTHRTCLQTFHNCR2SGIV3TFCB2BH2G2HVJ7ELRO3SDZA, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.903713268 (537h14m13.774796303s) [noise=true , address=145.79.227.41:4087 , piece_id=U6JCLCWNJDVHFOWWD3RTMOG5XYGUWXJORTV2BLKMF55B6L23FHMA, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.903887008 (537h14m13.774970044s) [noise=true , address=145.79.227.17:4971 , piece_id=UFXPZK7CFMXD7PSENRMMGI34U7JZDJFGSYZT7ZUJCESTHCHBIJBQ, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.904303001 (537h14m13.775386036s) [noise=true , address=145.79.227.3:4151 , piece_id=WISCT4SLYWN63FJ7OXB5HO7JOVJXQVDS6KICAQE3NQFBSVEJFKPQ, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.904620756 (537h14m13.775703821s) [noise=true , address=145.79.227.4:4185 , piece_id=UA5F6MOZG2L537H3DIJCRKJAEOQCGI2BMQGL4A7SLWTLFEAOMSUA, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.904877245 (537h14m13.77596028s) [noise=true , address=45.91.116.7:4010 , piece_id=PINGAXWR4WELATNHIQI7BNEGK4YZH7SEQZ6OBRI7XTEB2XMHAX3A, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.905002543 (537h14m13.776085578s) [noise=true , address=145.79.227.48:4301 , piece_id=DEONEQZHWS2RGAYCNK34WAH7TQXUN6CPXOLYGJDA6CEOFDEH4LLQ, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.905241498 (537h14m13.776324573s) [noise=true , address=45.91.116.13:4155 , piece_id=QBZZDRSWXDDOPRD2ZNO54S2UA7LRJZBWATIB57O5ETH4F2M7WQMA, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.905606823 (537h14m13.776689858s) [noise=true , address=45.91.116.8:4046 , piece_id=HTSRGPOSCX2WFOGS4GXJTPUTIRVZCQJUN4CG4QKLK3SJUC6AZLQA, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.905950037 (537h14m13.777033082s) [noise=true , address=77.90.31.15:4697 , piece_id=PUZ3ZS4DJK2OPM225MBMYTSGTY5LKSR74BQJET5YAUY6HDLTAS4A, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:15.623274913 (537h14m15.494357969s) [upload=471165]: Next returned nil segment Dec 28 00:43:15.623296375 (537h14m15.49437941s) [upload=471165]: Waiting for error group managing segments... Dec 28 00:43:15.907940263 (537h14m15.779023308s) [noise=true , address=145.79.227.17:4971 , piece_id=UFXPZK7CFMXD7PSENRMMGI34U7JZDJFGSYZT7ZUJCESTHCHBIJBQ, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:15.911507425 (537h14m15.78259047s) [noise=true , address=145.79.227.24:4583 , piece_id=ALFCBGFFGMCJIU6ULD36CMUKVI37W67UEWFS3HLSNCYHCXY6AJ7A, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:15.913055183 (537h14m15.784138258s) [noise=true , address=145.79.227.3:4151 , piece_id=WISCT4SLYWN63FJ7OXB5HO7JOVJXQVDS6KICAQE3NQFBSVEJFKPQ, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:15.931205985 (537h14m15.80228903s) [noise=true , address=145.79.227.41:4087 , piece_id=U6JCLCWNJDVHFOWWD3RTMOG5XYGUWXJORTV2BLKMF55B6L23FHMA, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:15.931599654 (537h14m15.802682689s) [noise=true , address=145.79.227.19:4077 , piece_id=DEEZHS5GIOTHRTCLQTFHNCR2SGIV3TFCB2BH2G2HVJ7ELRO3SDZA, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:15.936567945 (537h14m15.80765099s) [noise=true , address=145.79.227.48:4301 , piece_id=DEONEQZHWS2RGAYCNK34WAH7TQXUN6CPXOLYGJDA6CEOFDEH4LLQ, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:15.939257806 (537h14m15.810340841s) [noise=true , address=145.79.227.4:4185 , piece_id=UA5F6MOZG2L537H3DIJCRKJAEOQCGI2BMQGL4A7SLWTLFEAOMSUA, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:16.082116904 (537h14m15.953199939s) [noise=true , address=77.90.31.15:4697 , piece_id=PUZ3ZS4DJK2OPM225MBMYTSGTY5LKSR74BQJET5YAUY6HDLTAS4A, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:16.150063445 (537h14m16.02114649s) [noise=true , address=45.91.116.13:4155 , piece_id=QBZZDRSWXDDOPRD2ZNO54S2UA7LRJZBWATIB57O5ETH4F2M7WQMA, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:16.169105583 (537h14m16.040188628s) [noise=true , address=77.90.31.13:4629 , piece_id=HO4IKUBX4OVQI2OYEPVKZTPGIRWEQQ37PIYC2UDBTGSKTLQUOHKQ, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:16.194315498 (537h14m16.065398543s) [noise=true , address=45.91.116.7:4010 , piece_id=PINGAXWR4WELATNHIQI7BNEGK4YZH7SEQZ6OBRI7XTEB2XMHAX3A, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:16.220780552 (537h14m16.091863587s) [noise=true , address=45.91.116.8:4046 , piece_id=HTSRGPOSCX2WFOGS4GXJTPUTIRVZCQJUN4CG4QKLK3SJUC6AZLQA, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:16.839589454 (537h14m16.710672499s) [noise=true , address=77.90.31.11:4556 , piece_id=2EQZNKY2ZNWUGDB2TINXJEZTSPGVDDWHRJAHPKWNW3B7R6FRZHLA, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:16.839615473 (537h14m16.710698518s) [seg_pos={0 0}, upload=471165]: Segment reached optimal threshold of 13 pieces. Dec 28 00:43:16.839833949 (537h14m16.710916994s) [noise=true , address=45.91.116.11:4120 , piece_id=MAI5L5BM3XII7CSOIIWSXUNNSBYHWNOUWHWP3HK6S3Z4LAAMSHLQ, seg_pos={0 0}, upload=471165]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 28 00:43:16.839842255 (537h14m16.7109253s) [noise=true , address=45.91.116.11:4120 , piece_id=MAI5L5BM3XII7CSOIIWSXUNNSBYHWNOUWHWP3HK6S3Z4LAAMSHLQ, seg_pos={0 0}, upload=471165]: Long tail cancellation due to optimal threshold Dec 28 00:43:16.839971551 (537h14m16.711054586s) [noise=true , address=145.79.227.46:4603 , piece_id=G32YOVG6LJIDJMJETLHMTFGRSLRWIEUCIYPWUF6G27WSYOFQURMA, seg_pos={0 0}, upload=471165]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX): protocol: expected piece hash; context canceled; EOF Dec 28 00:43:16.840002039 (537h14m16.711085084s) [noise=true , address=145.79.227.46:4603 , piece_id=G32YOVG6LJIDJMJETLHMTFGRSLRWIEUCIYPWUF6G27WSYOFQURMA, seg_pos={0 0}, upload=471165]: Long tail cancellation due to optimal threshold Dec 28 00:43:16.840160532 (537h14m16.711243577s) [noise=true , address=45.91.116.12:4225 , piece_id=W2YMTTDYUV2WRCBAZG2U45FI3ISHOV5BLZDKNTLWNFFUYLIHCONA, seg_pos={0 0}, upload=471165]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 28 00:43:16.840187933 (537h14m16.711270968s) [noise=true , address=45.91.116.12:4225 , piece_id=W2YMTTDYUV2WRCBAZG2U45FI3ISHOV5BLZDKNTLWNFFUYLIHCONA, seg_pos={0 0}, upload=471165]: Long tail cancellation due to optimal threshold Dec 28 00:43:16.840206869 (537h14m16.711289915s) [seg_pos={0 0}, upload=471165]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:43:16.840440475 (537h14m16.71152352s) [upload=471165]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:43:16.840446817 (537h14m16.711529852s) [upload=471165]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:43:17.132569025 (537h14m17.00365206s) [upload=471165]: Done uploading segments... Dec 28 00:43:17.132587990 (537h14m17.003671036s) [upload=471165]: Upload finished. err: <nil> Duration: 8.695491051s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 34.914310818s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 00:43:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 34.914324865s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [X.X...X....X....X.XXX.X.X.XX.X.XXX...X.X.XX.XXX....X...X.X.X] duration: 30.004333366s trace: https://jaeger.dev.storj.io/trace/4b7e763d46d698c6 FAIL (2025-12-28 00:43:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: PUZ3ZS4DJK2OPM225MBMYTSGTY5LKSR74BQJET5YAUY6HDLTAS4A) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: 2EQZNKY2ZNWUGDB2TINXJEZTSPGVDDWHRJAHPKWNW3B7R6FRZHLA) context deadline exceeded ======LOGS====== Dec 28 00:43:13.657501310 (537h14m13.528584356s) [upload=471165]: Starting upload Dec 28 00:43:13.657618114 (537h14m13.528701149s) [upload=471165]: Done starting upload Dec 28 00:43:13.657904789 (537h14m13.528987834s) [upload=471165]: Uploading segments... Dec 28 00:43:13.657954253 (537h14m13.529037298s) [upload=471165]: Got next segment. Inline: false Dec 28 00:43:13.657987216 (537h14m13.529070261s) [upload=471165]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:43:13.657992906 (537h14m13.529075951s) [upload=471165]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:43:13.900922884 (537h14m13.772005919s) [seg_pos={0 0}, upload=471165]: Begin upload segment... Dec 28 00:43:13.900971277 (537h14m13.772054312s) [seg_pos={0 0}, upload=471165]: Done begin upload segment. Dec 28 00:43:13.901019599 (537h14m13.772102634s) [noise=true , address=145.79.227.46:4603 , piece_id=G32YOVG6LJIDJMJETLHMTFGRSLRWIEUCIYPWUF6G27WSYOFQURMA, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.901041701 (537h14m13.772124736s) [noise=true , address=145.79.227.24:4583 , piece_id=ALFCBGFFGMCJIU6ULD36CMUKVI37W67UEWFS3HLSNCYHCXY6AJ7A, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.901578393 (537h14m13.772661438s) [noise=true , address=45.91.116.11:4120 , piece_id=MAI5L5BM3XII7CSOIIWSXUNNSBYHWNOUWHWP3HK6S3Z4LAAMSHLQ, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.902086200 (537h14m13.773169245s) [noise=true , address=45.91.116.12:4225 , piece_id=W2YMTTDYUV2WRCBAZG2U45FI3ISHOV5BLZDKNTLWNFFUYLIHCONA, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.902229984 (537h14m13.773313019s) [noise=true , address=77.90.31.13:4629 , piece_id=HO4IKUBX4OVQI2OYEPVKZTPGIRWEQQ37PIYC2UDBTGSKTLQUOHKQ, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.903164243 (537h14m13.774247278s) [noise=true , address=77.90.31.11:4556 , piece_id=2EQZNKY2ZNWUGDB2TINXJEZTSPGVDDWHRJAHPKWNW3B7R6FRZHLA, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.903549527 (537h14m13.774632572s) [noise=true , address=145.79.227.19:4077 , piece_id=DEEZHS5GIOTHRTCLQTFHNCR2SGIV3TFCB2BH2G2HVJ7ELRO3SDZA, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.903713268 (537h14m13.774796303s) [noise=true , address=145.79.227.41:4087 , piece_id=U6JCLCWNJDVHFOWWD3RTMOG5XYGUWXJORTV2BLKMF55B6L23FHMA, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.903887008 (537h14m13.774970044s) [noise=true , address=145.79.227.17:4971 , piece_id=UFXPZK7CFMXD7PSENRMMGI34U7JZDJFGSYZT7ZUJCESTHCHBIJBQ, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.904303001 (537h14m13.775386036s) [noise=true , address=145.79.227.3:4151 , piece_id=WISCT4SLYWN63FJ7OXB5HO7JOVJXQVDS6KICAQE3NQFBSVEJFKPQ, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.904620756 (537h14m13.775703821s) [noise=true , address=145.79.227.4:4185 , piece_id=UA5F6MOZG2L537H3DIJCRKJAEOQCGI2BMQGL4A7SLWTLFEAOMSUA, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.904877245 (537h14m13.77596028s) [noise=true , address=45.91.116.7:4010 , piece_id=PINGAXWR4WELATNHIQI7BNEGK4YZH7SEQZ6OBRI7XTEB2XMHAX3A, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.905002543 (537h14m13.776085578s) [noise=true , address=145.79.227.48:4301 , piece_id=DEONEQZHWS2RGAYCNK34WAH7TQXUN6CPXOLYGJDA6CEOFDEH4LLQ, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.905241498 (537h14m13.776324573s) [noise=true , address=45.91.116.13:4155 , piece_id=QBZZDRSWXDDOPRD2ZNO54S2UA7LRJZBWATIB57O5ETH4F2M7WQMA, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.905606823 (537h14m13.776689858s) [noise=true , address=45.91.116.8:4046 , piece_id=HTSRGPOSCX2WFOGS4GXJTPUTIRVZCQJUN4CG4QKLK3SJUC6AZLQA, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:13.905950037 (537h14m13.777033082s) [noise=true , address=77.90.31.15:4697 , piece_id=PUZ3ZS4DJK2OPM225MBMYTSGTY5LKSR74BQJET5YAUY6HDLTAS4A, seg_pos={0 0}, upload=471165]: Uploading piece... Dec 28 00:43:15.623274913 (537h14m15.494357969s) [upload=471165]: Next returned nil segment Dec 28 00:43:15.623296375 (537h14m15.49437941s) [upload=471165]: Waiting for error group managing segments... Dec 28 00:43:15.907940263 (537h14m15.779023308s) [noise=true , address=145.79.227.17:4971 , piece_id=UFXPZK7CFMXD7PSENRMMGI34U7JZDJFGSYZT7ZUJCESTHCHBIJBQ, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:15.911507425 (537h14m15.78259047s) [noise=true , address=145.79.227.24:4583 , piece_id=ALFCBGFFGMCJIU6ULD36CMUKVI37W67UEWFS3HLSNCYHCXY6AJ7A, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:15.913055183 (537h14m15.784138258s) [noise=true , address=145.79.227.3:4151 , piece_id=WISCT4SLYWN63FJ7OXB5HO7JOVJXQVDS6KICAQE3NQFBSVEJFKPQ, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:15.931205985 (537h14m15.80228903s) [noise=true , address=145.79.227.41:4087 , piece_id=U6JCLCWNJDVHFOWWD3RTMOG5XYGUWXJORTV2BLKMF55B6L23FHMA, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:15.931599654 (537h14m15.802682689s) [noise=true , address=145.79.227.19:4077 , piece_id=DEEZHS5GIOTHRTCLQTFHNCR2SGIV3TFCB2BH2G2HVJ7ELRO3SDZA, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:15.936567945 (537h14m15.80765099s) [noise=true , address=145.79.227.48:4301 , piece_id=DEONEQZHWS2RGAYCNK34WAH7TQXUN6CPXOLYGJDA6CEOFDEH4LLQ, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:15.939257806 (537h14m15.810340841s) [noise=true , address=145.79.227.4:4185 , piece_id=UA5F6MOZG2L537H3DIJCRKJAEOQCGI2BMQGL4A7SLWTLFEAOMSUA, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:16.082116904 (537h14m15.953199939s) [noise=true , address=77.90.31.15:4697 , piece_id=PUZ3ZS4DJK2OPM225MBMYTSGTY5LKSR74BQJET5YAUY6HDLTAS4A, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:16.150063445 (537h14m16.02114649s) [noise=true , address=45.91.116.13:4155 , piece_id=QBZZDRSWXDDOPRD2ZNO54S2UA7LRJZBWATIB57O5ETH4F2M7WQMA, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:16.169105583 (537h14m16.040188628s) [noise=true , address=77.90.31.13:4629 , piece_id=HO4IKUBX4OVQI2OYEPVKZTPGIRWEQQ37PIYC2UDBTGSKTLQUOHKQ, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:16.194315498 (537h14m16.065398543s) [noise=true , address=45.91.116.7:4010 , piece_id=PINGAXWR4WELATNHIQI7BNEGK4YZH7SEQZ6OBRI7XTEB2XMHAX3A, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:16.220780552 (537h14m16.091863587s) [noise=true , address=45.91.116.8:4046 , piece_id=HTSRGPOSCX2WFOGS4GXJTPUTIRVZCQJUN4CG4QKLK3SJUC6AZLQA, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:16.839589454 (537h14m16.710672499s) [noise=true , address=77.90.31.11:4556 , piece_id=2EQZNKY2ZNWUGDB2TINXJEZTSPGVDDWHRJAHPKWNW3B7R6FRZHLA, seg_pos={0 0}, upload=471165]: Done uploading piece. err: <nil> Dec 28 00:43:16.839615473 (537h14m16.710698518s) [seg_pos={0 0}, upload=471165]: Segment reached optimal threshold of 13 pieces. Dec 28 00:43:16.839833949 (537h14m16.710916994s) [noise=true , address=45.91.116.11:4120 , piece_id=MAI5L5BM3XII7CSOIIWSXUNNSBYHWNOUWHWP3HK6S3Z4LAAMSHLQ, seg_pos={0 0}, upload=471165]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Dec 28 00:43:16.839842255 (537h14m16.7109253s) [noise=true , address=45.91.116.11:4120 , piece_id=MAI5L5BM3XII7CSOIIWSXUNNSBYHWNOUWHWP3HK6S3Z4LAAMSHLQ, seg_pos={0 0}, upload=471165]: Long tail cancellation due to optimal threshold Dec 28 00:43:16.839971551 (537h14m16.711054586s) [noise=true , address=145.79.227.46:4603 , piece_id=G32YOVG6LJIDJMJETLHMTFGRSLRWIEUCIYPWUF6G27WSYOFQURMA, seg_pos={0 0}, upload=471165]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ssbeqxAWdq216V5ukNXZ3zvRroDf1x2hSs8cLtepUpefugtkX): protocol: expected piece hash; context canceled; EOF Dec 28 00:43:16.840002039 (537h14m16.711085084s) [noise=true , address=145.79.227.46:4603 , piece_id=G32YOVG6LJIDJMJETLHMTFGRSLRWIEUCIYPWUF6G27WSYOFQURMA, seg_pos={0 0}, upload=471165]: Long tail cancellation due to optimal threshold Dec 28 00:43:16.840160532 (537h14m16.711243577s) [noise=true , address=45.91.116.12:4225 , piece_id=W2YMTTDYUV2WRCBAZG2U45FI3ISHOV5BLZDKNTLWNFFUYLIHCONA, seg_pos={0 0}, upload=471165]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 28 00:43:16.840187933 (537h14m16.711270968s) [noise=true , address=45.91.116.12:4225 , piece_id=W2YMTTDYUV2WRCBAZG2U45FI3ISHOV5BLZDKNTLWNFFUYLIHCONA, seg_pos={0 0}, upload=471165]: Long tail cancellation due to optimal threshold Dec 28 00:43:16.840206869 (537h14m16.711289915s) [seg_pos={0 0}, upload=471165]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:43:16.840440475 (537h14m16.71152352s) [upload=471165]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:43:16.840446817 (537h14m16.711529852s) [upload=471165]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:43:17.132569025 (537h14m17.00365206s) [upload=471165]: Done uploading segments... Dec 28 00:43:17.132587990 (537h14m17.003671036s) [upload=471165]: Upload finished. err: <nil> Duration: 30.004358254s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 31.496163182s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 00:43:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 31.496177971s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 41.971695538s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 00:42:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 41.971731126s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.X.X...X....X....X.XXX.X.X.XX.X.XXX...X.X.XX.XXX....X...X.X.] duration: 6.086376542s trace: https://jaeger.dev.storj.io/trace/25b3560de7408c2d FAIL (2025-12-28 00:42:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: I3O2NXBFN5BSI7VOABZNEEEJHVE5AOQODSGA5BVWT6T2ZY7EGRAQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: EQLQSSI3BJZJ32ZMLIPB3GUOLADWZCUJNJCQDKO7PKXCPFHLRJXQ) context deadline exceeded ======LOGS====== 00:41:55.294043886 (537h12m55.165126921s) [upload=471143]: Starting upload Dec 28 00:41:55.294098801 (537h12m55.165181836s) [upload=471143]: Done starting upload Dec 28 00:41:55.294192248 (537h12m55.165275293s) [upload=471143]: Uploading segments... Dec 28 00:41:55.294343206 (537h12m55.165426251s) [upload=471143]: Got next segment. Inline: false Dec 28 00:41:55.294552544 (537h12m55.16563559s) [upload=471143]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:41:55.294561982 (537h12m55.165645017s) [upload=471143]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:41:55.544564727 (537h12m55.415647782s) [seg_pos={0 0}, upload=471143]: Begin upload segment... Dec 28 00:41:55.544625292 (537h12m55.415708337s) [seg_pos={0 0}, upload=471143]: Done begin upload segment. Dec 28 00:41:55.544681168 (537h12m55.415764213s) [noise=true , address=77.90.31.11:4555 , piece_id=YMXOBMSTXMFMLN73HP4UDHOGUFD2NJI55VFFSKIHPAB6Q5BXXWOQ, seg_pos={0 0}, upload=471143]: Uploading piece... Dec 28 00:41:55.544697410 (537h12m55.415780445s) [noise=true , address=145.79.227.15:4873 , piece_id=GINOUOFL5KA4WE7VHLQAZTHTLDK3EOBOGHZCFTOM7PNBBXVKWCEA, seg_pos={0 0}, upload=471143]: Uploading piece... Dec 28 00:41:55.546227343 (537h12m55.417310388s) [noise=true , address=145.79.227.19:4078 , piece_id=GYUBBNH5BM26D3OBQI4PGNW7S4554BCNQUBZA6WHC45ESEZNXKXA, seg_pos={0 0}, upload=471143]: Uploading piece... Dec 28 00:41:55.546697929 (537h12m55.417780964s) [noise=true , address=145.79.227.3:4150 , piece_id=4QPK3WYGWUBOBTCJCIFUCKC6UKTG3Y2RUJM6NTNA3CUHFMZRDOEQ, seg_pos={0 0}, upload=471143]: Uploading piece... Dec 28 00:41:55.547072582 (537h12m55.418155627s) [noise=true , address=145.79.227.38:4977 , piece_id=SLXM7TEYCRNWCXEXL5566VXXB3TCJSPWUSJPCDWE5NFIQV6JN5XQ, seg_pos={0 0}, upload=471143]: Uploading piece... Dec 28 00:41:55.547422609 (537h12m55.418505644s) [noise=true , address=145.79.227.46:4602 , piece_id=KL34MTMCAYI3GWDW52B7EEWFPATGYXOEJKBNKAYABOB7WU4ETBHA, seg_pos={0 0}, upload=471143]: Uploading piece... Dec 28 00:41:55.548335717 (537h12m55.419418772s) [noise=true , address=45.91.116.9:4264 , piece_id=JQOCUS4UW2AZIZZDQBDCISC3423ZQZGWYOH3TAHNCALJCYCGFPVQ, seg_pos={0 0}, upload=471143]: Uploading piece... Dec 28 00:41:55.548647371 (537h12m55.419730426s) [noise=true , address=145.79.227.9:4365 , piece_id=PPH4FLLWU5SWIG473KPH3UNSMHCZ67KLEPWC5ZAMXEJPW7BXS5EA, seg_pos={0 0}, upload=471143]: Uploading piece... Dec 28 00:41:55.549540281 (537h12m55.420623326s) [noise=true , address=145.79.227.11:4729 , piece_id=PJ6EN24HHVB5L5YP6EZWKRNLEWCJA27IPF4JBZF2GRVSVROEKQ7A, seg_pos={0 0}, upload=471143]: Uploading piece... Dec 28 00:41:55.549650210 (537h12m55.420733245s) [noise=true , address=145.79.227.36:4519 , piece_id=JPH6JSN7UVDC6SOHK6AFFZLCWNIJ2N7Z2CKXF35EN532D5AWU6DA, seg_pos={0 0}, upload=471143]: Uploading piece... Dec 28 00:41:55.550561515 (537h12m55.421644561s) [noise=true , address=45.91.116.12:4225 , piece_id=I3O2NXBFN5BSI7VOABZNEEEJHVE5AOQODSGA5BVWT6T2ZY7EGRAQ, seg_pos={0 0}, upload=471143]: Uploading piece... Dec 28 00:41:55.551022122 (537h12m55.422105167s) [noise=true , address=145.79.227.42:4121 , piece_id=7YPYZXLUXTLSQWRLLIWPZRUWEVEZCULGWGTUV6ZO67NULFYB3MRQ, seg_pos={0 0}, upload=471143]: Uploading piece... Dec 28 00:41:55.551554757 (537h12m55.422637792s) [noise=true , address=45.91.116.7:4010 , piece_id=EQLQSSI3BJZJ32ZMLIPB3GUOLADWZCUJNJCQDKO7PKXCPFHLRJXQ, seg_pos={0 0}, upload=471143]: Uploading piece... Dec 28 00:41:55.552457015 (537h12m55.42354006s) [noise=true , address=145.79.227.45:4231 , piece_id=UAGFNYSRMGRHRQAN6RWGSS5GOEQEW72AUFL7D7AWF64TEY4S3IZA, seg_pos={0 0}, upload=471143]: Uploading piece... Dec 28 00:41:55.553766428 (537h12m55.424849473s) [noise=true , address=45.91.116.6:4973 , piece_id=ZUPMI3GSBL2S52W7NBXJSW3OAKQGV7MYJIANX7K3QFOOSVKBP5SA, seg_pos={0 0}, upload=471143]: Uploading piece... Dec 28 00:41:55.554336273 (537h12m55.425419318s) [noise=true , address=45.91.116.4:4937 , piece_id=UZPUQZNUHPS4SF6NSX5HLQDRZ2CQ7EQ3DYGFTRL4GLHVUINBKKFA, seg_pos={0 0}, upload=471143]: Uploading piece... Dec 28 00:41:57.263456868 (537h12m57.134539903s) [upload=471143]: Next returned nil segment Dec 28 00:41:57.263476294 (537h12m57.13455934s) [upload=471143]: Waiting for error group managing segments... Dec 28 00:41:57.542623235 (537h12m57.413706281s) [noise=true , address=145.79.227.11:4729 , piece_id=PJ6EN24HHVB5L5YP6EZWKRNLEWCJA27IPF4JBZF2GRVSVROEKQ7A, seg_pos={0 0}, upload=471143]: Done uploading piece. err: <nil> Dec 28 00:41:57.557706015 (537h12m57.42878905s) [noise=true , address=145.79.227.36:4519 , piece_id=JPH6JSN7UVDC6SOHK6AFFZLCWNIJ2N7Z2CKXF35EN532D5AWU6DA, seg_pos={0 0}, upload=471143]: Done uploading piece. err: <nil> Dec 28 00:41:57.563449582 (537h12m57.434532617s) [noise=true , address=145.79.227.15:4873 , piece_id=GINOUOFL5KA4WE7VHLQAZTHTLDK3EOBOGHZCFTOM7PNBBXVKWCEA, seg_pos={0 0}, upload=471143]: Done uploading piece. err: <nil> Dec 28 00:41:57.565995931 (537h12m57.437078966s) [noise=true , address=145.79.227.38:4977 , piece_id=SLXM7TEYCRNWCXEXL5566VXXB3TCJSPWUSJPCDWE5NFIQV6JN5XQ, seg_pos={0 0}, upload=471143]: Done uploading piece. err: <nil> Dec 28 00:41:57.571086986 (537h12m57.442170031s) [noise=true , address=145.79.227.46:4602 , piece_id=KL34MTMCAYI3GWDW52B7EEWFPATGYXOEJKBNKAYABOB7WU4ETBHA, seg_pos={0 0}, upload=471143]: Done uploading piece. err: <nil> Dec 28 00:41:57.575368137 (537h12m57.446451172s) [noise=true , address=145.79.227.9:4365 , piece_id=PPH4FLLWU5SWIG473KPH3UNSMHCZ67KLEPWC5ZAMXEJPW7BXS5EA, seg_pos={0 0}, upload=471143]: Done uploading piece. err: <nil> Dec 28 00:41:57.576278581 (537h12m57.447361626s) [noise=true , address=145.79.227.45:4231 , piece_id=UAGFNYSRMGRHRQAN6RWGSS5GOEQEW72AUFL7D7AWF64TEY4S3IZA, seg_pos={0 0}, upload=471143]: Done uploading piece. err: <nil> Dec 28 00:41:57.577078865 (537h12m57.44816191s) [noise=true , address=145.79.227.42:4121 , piece_id=7YPYZXLUXTLSQWRLLIWPZRUWEVEZCULGWGTUV6ZO67NULFYB3MRQ, seg_pos={0 0}, upload=471143]: Done uploading piece. err: <nil> Dec 28 00:41:57.730526354 (537h12m57.601609389s) [noise=true , address=77.90.31.11:4555 , piece_id=YMXOBMSTXMFMLN73HP4UDHOGUFD2NJI55VFFSKIHPAB6Q5BXXWOQ, seg_pos={0 0}, upload=471143]: Done uploading piece. err: <nil> Dec 28 00:41:57.813630338 (537h12m57.684713373s) [noise=true , address=45.91.116.7:4010 , piece_id=EQLQSSI3BJZJ32ZMLIPB3GUOLADWZCUJNJCQDKO7PKXCPFHLRJXQ, seg_pos={0 0}, upload=471143]: Done uploading piece. err: <nil> Dec 28 00:41:57.888450877 (537h12m57.759533923s) [noise=true , address=45.91.116.9:4264 , piece_id=JQOCUS4UW2AZIZZDQBDCISC3423ZQZGWYOH3TAHNCALJCYCGFPVQ, seg_pos={0 0}, upload=471143]: Done uploading piece. err: <nil> Dec 28 00:41:58.002822529 (537h12m57.873905564s) [noise=true , address=45.91.116.12:4225 , piece_id=I3O2NXBFN5BSI7VOABZNEEEJHVE5AOQODSGA5BVWT6T2ZY7EGRAQ, seg_pos={0 0}, upload=471143]: Done uploading piece. err: <nil> Dec 28 00:41:58.497617180 (537h12m58.368700275s) [noise=true , address=45.91.116.4:4937 , piece_id=UZPUQZNUHPS4SF6NSX5HLQDRZ2CQ7EQ3DYGFTRL4GLHVUINBKKFA, seg_pos={0 0}, upload=471143]: Done uploading piece. err: <nil> Dec 28 00:41:58.497641366 (537h12m58.368724411s) [seg_pos={0 0}, upload=471143]: Segment reached optimal threshold of 13 pieces. Dec 28 00:41:58.497907222 (537h12m58.368990267s) [noise=true , address=45.91.116.6:4973 , piece_id=ZUPMI3GSBL2S52W7NBXJSW3OAKQGV7MYJIANX7K3QFOOSVKBP5SA, seg_pos={0 0}, upload=471143]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 28 00:41:58.497917662 (537h12m58.369000708s) [noise=true , address=45.91.116.6:4973 , piece_id=ZUPMI3GSBL2S52W7NBXJSW3OAKQGV7MYJIANX7K3QFOOSVKBP5SA, seg_pos={0 0}, upload=471143]: Long tail cancellation due to optimal threshold Dec 28 00:41:58.497932961 (537h12m58.369016006s) [noise=true , address=145.79.227.19:4078 , piece_id=GYUBBNH5BM26D3OBQI4PGNW7S4554BCNQUBZA6WHC45ESEZNXKXA, seg_pos={0 0}, upload=471143]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:41846->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:41846->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41846->145.79.227.19:4078: use of closed network connection Dec 28 00:41:58.497945334 (537h12m58.369028379s) [noise=true , address=145.79.227.19:4078 , piece_id=GYUBBNH5BM26D3OBQI4PGNW7S4554BCNQUBZA6WHC45ESEZNXKXA, seg_pos={0 0}, upload=471143]: Long tail cancellation due to optimal threshold Dec 28 00:41:58.498052428 (537h12m58.369135473s) [noise=true , address=145.79.227.3:4150 , piece_id=4QPK3WYGWUBOBTCJCIFUCKC6UKTG3Y2RUJM6NTNA3CUHFMZRDOEQ, seg_pos={0 0}, upload=471143]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Dec 28 00:41:58.498059853 (537h12m58.369142898s) [noise=true , address=145.79.227.3:4150 , piece_id=4QPK3WYGWUBOBTCJCIFUCKC6UKTG3Y2RUJM6NTNA3CUHFMZRDOEQ, seg_pos={0 0}, upload=471143]: Long tail cancellation due to optimal threshold Dec 28 00:41:58.498082125 (537h12m58.36916516s) [seg_pos={0 0}, upload=471143]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:41:58.498168951 (537h12m58.369251996s) [upload=471143]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:41:58.498175162 (537h12m58.369258207s) [upload=471143]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:41:58.793973069 (537h12m58.665056114s) [upload=471143]: Done uploading segments... Dec 28 00:41:58.793991975 (537h12m58.66507501s) [upload=471143]: Upload finished. err: <nil> Duration: 6.086391331s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [..X.X...X....X....X.XXX.X.X.XX.X.XXX...X.X.XX.XXX....X...X.X] duration: 30.003401837s trace: https://jaeger.dev.storj.io/trace/25b3560de7408c2d FAIL (2025-12-28 00:42:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: I3O2NXBFN5BSI7VOABZNEEEJHVE5AOQODSGA5BVWT6T2ZY7EGRAQ) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: EQLQSSI3BJZJ32ZMLIPB3GUOLADWZCUJNJCQDKO7PKXCPFHLRJXQ) context deadline exceeded ======LOGS====== 00:41:55.294043886 (537h12m55.165126921s) [upload=471143]: Starting upload Dec 28 00:41:55.294098801 (537h12m55.165181836s) [upload=471143]: Done starting upload Dec 28 00:41:55.294192248 (537h12m55.165275293s) [upload=471143]: Uploading segments... Dec 28 00:41:55.294343206 (537h12m55.165426251s) [upload=471143]: Got next segment. Inline: false Dec 28 00:41:55.294552544 (537h12m55.16563559s) [upload=471143]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:41:55.294561982 (537h12m55.165645017s) [upload=471143]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:41:55.544564727 (537h12m55.415647782s) [seg_pos={0 0}, upload=471143]: Begin upload segment... Dec 28 00:41:55.544625292 (537h12m55.415708337s) [seg_pos={0 0}, upload=471143]: Done begin upload segment. Dec 28 00:41:55.544681168 (537h12m55.415764213s) [noise=true , address=77.90.31.11:4555 , piece_id=YMXOBMSTXMFMLN73HP4UDHOGUFD2NJI55VFFSKIHPAB6Q5BXXWOQ, seg_pos={0 0}, upload=471143]: Uploading piece... Dec 28 00:41:55.544697410 (537h12m55.415780445s) [noise=true , address=145.79.227.15:4873 , piece_id=GINOUOFL5KA4WE7VHLQAZTHTLDK3EOBOGHZCFTOM7PNBBXVKWCEA, seg_pos={0 0}, upload=471143]: Uploading piece... Dec 28 00:41:55.546227343 (537h12m55.417310388s) [noise=true , address=145.79.227.19:4078 , piece_id=GYUBBNH5BM26D3OBQI4PGNW7S4554BCNQUBZA6WHC45ESEZNXKXA, seg_pos={0 0}, upload=471143]: Uploading piece... Dec 28 00:41:55.546697929 (537h12m55.417780964s) [noise=true , address=145.79.227.3:4150 , piece_id=4QPK3WYGWUBOBTCJCIFUCKC6UKTG3Y2RUJM6NTNA3CUHFMZRDOEQ, seg_pos={0 0}, upload=471143]: Uploading piece... Dec 28 00:41:55.547072582 (537h12m55.418155627s) [noise=true , address=145.79.227.38:4977 , piece_id=SLXM7TEYCRNWCXEXL5566VXXB3TCJSPWUSJPCDWE5NFIQV6JN5XQ, seg_pos={0 0}, upload=471143]: Uploading piece... Dec 28 00:41:55.547422609 (537h12m55.418505644s) [noise=true , address=145.79.227.46:4602 , piece_id=KL34MTMCAYI3GWDW52B7EEWFPATGYXOEJKBNKAYABOB7WU4ETBHA, seg_pos={0 0}, upload=471143]: Uploading piece... Dec 28 00:41:55.548335717 (537h12m55.419418772s) [noise=true , address=45.91.116.9:4264 , piece_id=JQOCUS4UW2AZIZZDQBDCISC3423ZQZGWYOH3TAHNCALJCYCGFPVQ, seg_pos={0 0}, upload=471143]: Uploading piece... Dec 28 00:41:55.548647371 (537h12m55.419730426s) [noise=true , address=145.79.227.9:4365 , piece_id=PPH4FLLWU5SWIG473KPH3UNSMHCZ67KLEPWC5ZAMXEJPW7BXS5EA, seg_pos={0 0}, upload=471143]: Uploading piece... Dec 28 00:41:55.549540281 (537h12m55.420623326s) [noise=true , address=145.79.227.11:4729 , piece_id=PJ6EN24HHVB5L5YP6EZWKRNLEWCJA27IPF4JBZF2GRVSVROEKQ7A, seg_pos={0 0}, upload=471143]: Uploading piece... Dec 28 00:41:55.549650210 (537h12m55.420733245s) [noise=true , address=145.79.227.36:4519 , piece_id=JPH6JSN7UVDC6SOHK6AFFZLCWNIJ2N7Z2CKXF35EN532D5AWU6DA, seg_pos={0 0}, upload=471143]: Uploading piece... Dec 28 00:41:55.550561515 (537h12m55.421644561s) [noise=true , address=45.91.116.12:4225 , piece_id=I3O2NXBFN5BSI7VOABZNEEEJHVE5AOQODSGA5BVWT6T2ZY7EGRAQ, seg_pos={0 0}, upload=471143]: Uploading piece... Dec 28 00:41:55.551022122 (537h12m55.422105167s) [noise=true , address=145.79.227.42:4121 , piece_id=7YPYZXLUXTLSQWRLLIWPZRUWEVEZCULGWGTUV6ZO67NULFYB3MRQ, seg_pos={0 0}, upload=471143]: Uploading piece... Dec 28 00:41:55.551554757 (537h12m55.422637792s) [noise=true , address=45.91.116.7:4010 , piece_id=EQLQSSI3BJZJ32ZMLIPB3GUOLADWZCUJNJCQDKO7PKXCPFHLRJXQ, seg_pos={0 0}, upload=471143]: Uploading piece... Dec 28 00:41:55.552457015 (537h12m55.42354006s) [noise=true , address=145.79.227.45:4231 , piece_id=UAGFNYSRMGRHRQAN6RWGSS5GOEQEW72AUFL7D7AWF64TEY4S3IZA, seg_pos={0 0}, upload=471143]: Uploading piece... Dec 28 00:41:55.553766428 (537h12m55.424849473s) [noise=true , address=45.91.116.6:4973 , piece_id=ZUPMI3GSBL2S52W7NBXJSW3OAKQGV7MYJIANX7K3QFOOSVKBP5SA, seg_pos={0 0}, upload=471143]: Uploading piece... Dec 28 00:41:55.554336273 (537h12m55.425419318s) [noise=true , address=45.91.116.4:4937 , piece_id=UZPUQZNUHPS4SF6NSX5HLQDRZ2CQ7EQ3DYGFTRL4GLHVUINBKKFA, seg_pos={0 0}, upload=471143]: Uploading piece... Dec 28 00:41:57.263456868 (537h12m57.134539903s) [upload=471143]: Next returned nil segment Dec 28 00:41:57.263476294 (537h12m57.13455934s) [upload=471143]: Waiting for error group managing segments... Dec 28 00:41:57.542623235 (537h12m57.413706281s) [noise=true , address=145.79.227.11:4729 , piece_id=PJ6EN24HHVB5L5YP6EZWKRNLEWCJA27IPF4JBZF2GRVSVROEKQ7A, seg_pos={0 0}, upload=471143]: Done uploading piece. err: <nil> Dec 28 00:41:57.557706015 (537h12m57.42878905s) [noise=true , address=145.79.227.36:4519 , piece_id=JPH6JSN7UVDC6SOHK6AFFZLCWNIJ2N7Z2CKXF35EN532D5AWU6DA, seg_pos={0 0}, upload=471143]: Done uploading piece. err: <nil> Dec 28 00:41:57.563449582 (537h12m57.434532617s) [noise=true , address=145.79.227.15:4873 , piece_id=GINOUOFL5KA4WE7VHLQAZTHTLDK3EOBOGHZCFTOM7PNBBXVKWCEA, seg_pos={0 0}, upload=471143]: Done uploading piece. err: <nil> Dec 28 00:41:57.565995931 (537h12m57.437078966s) [noise=true , address=145.79.227.38:4977 , piece_id=SLXM7TEYCRNWCXEXL5566VXXB3TCJSPWUSJPCDWE5NFIQV6JN5XQ, seg_pos={0 0}, upload=471143]: Done uploading piece. err: <nil> Dec 28 00:41:57.571086986 (537h12m57.442170031s) [noise=true , address=145.79.227.46:4602 , piece_id=KL34MTMCAYI3GWDW52B7EEWFPATGYXOEJKBNKAYABOB7WU4ETBHA, seg_pos={0 0}, upload=471143]: Done uploading piece. err: <nil> Dec 28 00:41:57.575368137 (537h12m57.446451172s) [noise=true , address=145.79.227.9:4365 , piece_id=PPH4FLLWU5SWIG473KPH3UNSMHCZ67KLEPWC5ZAMXEJPW7BXS5EA, seg_pos={0 0}, upload=471143]: Done uploading piece. err: <nil> Dec 28 00:41:57.576278581 (537h12m57.447361626s) [noise=true , address=145.79.227.45:4231 , piece_id=UAGFNYSRMGRHRQAN6RWGSS5GOEQEW72AUFL7D7AWF64TEY4S3IZA, seg_pos={0 0}, upload=471143]: Done uploading piece. err: <nil> Dec 28 00:41:57.577078865 (537h12m57.44816191s) [noise=true , address=145.79.227.42:4121 , piece_id=7YPYZXLUXTLSQWRLLIWPZRUWEVEZCULGWGTUV6ZO67NULFYB3MRQ, seg_pos={0 0}, upload=471143]: Done uploading piece. err: <nil> Dec 28 00:41:57.730526354 (537h12m57.601609389s) [noise=true , address=77.90.31.11:4555 , piece_id=YMXOBMSTXMFMLN73HP4UDHOGUFD2NJI55VFFSKIHPAB6Q5BXXWOQ, seg_pos={0 0}, upload=471143]: Done uploading piece. err: <nil> Dec 28 00:41:57.813630338 (537h12m57.684713373s) [noise=true , address=45.91.116.7:4010 , piece_id=EQLQSSI3BJZJ32ZMLIPB3GUOLADWZCUJNJCQDKO7PKXCPFHLRJXQ, seg_pos={0 0}, upload=471143]: Done uploading piece. err: <nil> Dec 28 00:41:57.888450877 (537h12m57.759533923s) [noise=true , address=45.91.116.9:4264 , piece_id=JQOCUS4UW2AZIZZDQBDCISC3423ZQZGWYOH3TAHNCALJCYCGFPVQ, seg_pos={0 0}, upload=471143]: Done uploading piece. err: <nil> Dec 28 00:41:58.002822529 (537h12m57.873905564s) [noise=true , address=45.91.116.12:4225 , piece_id=I3O2NXBFN5BSI7VOABZNEEEJHVE5AOQODSGA5BVWT6T2ZY7EGRAQ, seg_pos={0 0}, upload=471143]: Done uploading piece. err: <nil> Dec 28 00:41:58.497617180 (537h12m58.368700275s) [noise=true , address=45.91.116.4:4937 , piece_id=UZPUQZNUHPS4SF6NSX5HLQDRZ2CQ7EQ3DYGFTRL4GLHVUINBKKFA, seg_pos={0 0}, upload=471143]: Done uploading piece. err: <nil> Dec 28 00:41:58.497641366 (537h12m58.368724411s) [seg_pos={0 0}, upload=471143]: Segment reached optimal threshold of 13 pieces. Dec 28 00:41:58.497907222 (537h12m58.368990267s) [noise=true , address=45.91.116.6:4973 , piece_id=ZUPMI3GSBL2S52W7NBXJSW3OAKQGV7MYJIANX7K3QFOOSVKBP5SA, seg_pos={0 0}, upload=471143]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF Dec 28 00:41:58.497917662 (537h12m58.369000708s) [noise=true , address=45.91.116.6:4973 , piece_id=ZUPMI3GSBL2S52W7NBXJSW3OAKQGV7MYJIANX7K3QFOOSVKBP5SA, seg_pos={0 0}, upload=471143]: Long tail cancellation due to optimal threshold Dec 28 00:41:58.497932961 (537h12m58.369016006s) [noise=true , address=145.79.227.19:4078 , piece_id=GYUBBNH5BM26D3OBQI4PGNW7S4554BCNQUBZA6WHC45ESEZNXKXA, seg_pos={0 0}, upload=471143]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:41846->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:41846->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41846->145.79.227.19:4078: use of closed network connection Dec 28 00:41:58.497945334 (537h12m58.369028379s) [noise=true , address=145.79.227.19:4078 , piece_id=GYUBBNH5BM26D3OBQI4PGNW7S4554BCNQUBZA6WHC45ESEZNXKXA, seg_pos={0 0}, upload=471143]: Long tail cancellation due to optimal threshold Dec 28 00:41:58.498052428 (537h12m58.369135473s) [noise=true , address=145.79.227.3:4150 , piece_id=4QPK3WYGWUBOBTCJCIFUCKC6UKTG3Y2RUJM6NTNA3CUHFMZRDOEQ, seg_pos={0 0}, upload=471143]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: expected piece hash; context canceled; EOF Dec 28 00:41:58.498059853 (537h12m58.369142898s) [noise=true , address=145.79.227.3:4150 , piece_id=4QPK3WYGWUBOBTCJCIFUCKC6UKTG3Y2RUJM6NTNA3CUHFMZRDOEQ, seg_pos={0 0}, upload=471143]: Long tail cancellation due to optimal threshold Dec 28 00:41:58.498082125 (537h12m58.36916516s) [seg_pos={0 0}, upload=471143]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:41:58.498168951 (537h12m58.369251996s) [upload=471143]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:41:58.498175162 (537h12m58.369258207s) [upload=471143]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:41:58.793973069 (537h12m58.665056114s) [upload=471143]: Done uploading segments... Dec 28 00:41:58.793991975 (537h12m58.66507501s) [upload=471143]: Upload finished. err: <nil> Duration: 30.00341481s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [...X.X...X....X....X.XXX.X.X.XX.X.XXX...X.X.XX.XXX....X...X.] duration: 28.515074503s trace: https://jaeger.dev.storj.io/trace/42c1eb9479fa933b FAIL (2025-12-28 00:41:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: 7FFH3Q6MFSZAF42W6NBJ36P655RQML66A6MKHHBXCSKELC5LRPRA) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: XP5TQTIW7GV5J6B2XOIGEDRCFAZ56FT2Z7IIN2E4RJUDZHRRAXLQ) context deadline exceeded ======LOGS====== gments... Dec 28 00:40:55.284980149 (537h11m55.156063184s) [upload=471124]: Got next segment. Inline: false Dec 28 00:40:55.285025015 (537h11m55.15610808s) [upload=471124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:40:55.285030625 (537h11m55.15611367s) [upload=471124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:40:55.531004529 (537h11m55.402087564s) [seg_pos={0 0}, upload=471124]: Begin upload segment... Dec 28 00:40:55.531052491 (537h11m55.402135526s) [seg_pos={0 0}, upload=471124]: Done begin upload segment. Dec 28 00:40:55.531079974 (537h11m55.402163009s) [noise=true , address=145.79.227.15:4874 , piece_id=G4QDIAFCWDKAJYE4FXLSCGWXDC55CPYZJLZ5XYU6AQCT334BXZEQ, seg_pos={0 0}, upload=471124]: Uploading piece... Dec 28 00:40:55.531185635 (537h11m55.40226868s) [noise=true , address=145.79.227.7:4293 , piece_id=HEEHAFPHEG4FU4O43MW7WRSBMMQYMSJ2E6QGBYHIVH2CFTPLKIKQ, seg_pos={0 0}, upload=471124]: Uploading piece... Dec 28 00:40:55.531955240 (537h11m55.403038275s) [noise=true , address=145.79.227.52:4446 , piece_id=6LJCHVETPYZYJ5CLXFFXQEDKYZTUEAFB7NTJPC2MI3PEKKUVEPFA, seg_pos={0 0}, upload=471124]: Uploading piece... Dec 28 00:40:55.532608002 (537h11m55.403691047s) [noise=true , address=45.91.116.6:4976 , piece_id=YDY6JM7TPX2ODNQ6C4MAUY2H5XWLYIYRQZTXSEGM4EL2AQNOLXLQ, seg_pos={0 0}, upload=471124]: Uploading piece... Dec 28 00:40:55.533163520 (537h11m55.404246555s) [noise=true , address=45.91.116.11:4120 , piece_id=XP5TQTIW7GV5J6B2XOIGEDRCFAZ56FT2Z7IIN2E4RJUDZHRRAXLQ, seg_pos={0 0}, upload=471124]: Uploading piece... Dec 28 00:40:55.533410761 (537h11m55.404493816s) [noise=true , address=145.79.227.11:4730 , piece_id=M2UQ5OLQJEBDUVZU3TVDPLQQ7MHZEDO4FGEVPFJACKQLBMA6EKOQ, seg_pos={0 0}, upload=471124]: Uploading piece... Dec 28 00:40:55.534007718 (537h11m55.405090754s) [noise=true , address=145.79.227.38:4979 , piece_id=PWWIPBWNFMYYPDQBLCPZVYJAHUERO7LQGYGOY77MLSCUT2FFXHGQ, seg_pos={0 0}, upload=471124]: Uploading piece... Dec 28 00:40:55.534125793 (537h11m55.405208828s) [noise=true , address=145.79.227.16:4909 , piece_id=CUI5BXLCKAF6LYS5PVYD43SALWFUS4X7J5VIXXMBBPZK2H2EEBBQ, seg_pos={0 0}, upload=471124]: Uploading piece... Dec 28 00:40:55.534435914 (537h11m55.405518949s) [noise=true , address=145.79.227.50:4373 , piece_id=3TI2DVFZABU4DGK74ONBWULNM45MNNQMSQVUWJP7WSY45XKN76UQ, seg_pos={0 0}, upload=471124]: Uploading piece... Dec 28 00:40:55.534760822 (537h11m55.405843857s) [noise=true , address=45.91.116.14:4190 , piece_id=7FFH3Q6MFSZAF42W6NBJ36P655RQML66A6MKHHBXCSKELC5LRPRA, seg_pos={0 0}, upload=471124]: Uploading piece... Dec 28 00:40:55.534837488 (537h11m55.405920523s) [noise=true , address=45.91.116.7:4009 , piece_id=EPEVKHVQLF2BRZWX3HVPOO43YNABEHYAIXGJLBUFL6NDA7NM7BOA, seg_pos={0 0}, upload=471124]: Uploading piece... Dec 28 00:40:55.535459903 (537h11m55.406542938s) [noise=true , address=145.79.227.42:4123 , piece_id=CW26NMV2KRN2MUCYTSA3Z4VXMGHS25XLFHCOTBU2HXJJXIZCI75Q, seg_pos={0 0}, upload=471124]: Uploading piece... Dec 28 00:40:55.536077189 (537h11m55.407160234s) [noise=true , address=145.79.227.45:4230 , piece_id=5EQL36BUYUYYLJE5HCVNHS4YJ3IVZ5DEBWLBOGMMJ6T5IP2HTCSQ, seg_pos={0 0}, upload=471124]: Uploading piece... Dec 28 00:40:55.536492960 (537h11m55.407576005s) [noise=true , address=145.79.227.35:4481 , piece_id=S3AXSPCHPWF276YJ7WMDCJKHNPL5SA3PY37YUVTDGVDM2RDX76ZA, seg_pos={0 0}, upload=471124]: Uploading piece... Dec 28 00:40:55.536878545 (537h11m55.40796159s) [noise=true , address=145.79.227.36:4517 , piece_id=35DVJKKSJOSLSPFNQHEM35GY6DWW7REUHISCJMZUQFILS2RKML2A, seg_pos={0 0}, upload=471124]: Uploading piece... Dec 28 00:40:55.541585487 (537h11m55.412668532s) [noise=true , address=145.79.227.41:4087 , piece_id=ABG5YAGRUZ3DRBBYPU6LGJT6UWZLRXG4TC7RAO3ASCAU6VTZO3NQ, seg_pos={0 0}, upload=471124]: Uploading piece... Dec 28 00:40:57.155294232 (537h11m57.026377277s) [upload=471124]: Next returned nil segment Dec 28 00:40:57.155313148 (537h11m57.026396203s) [upload=471124]: Waiting for error group managing segments... Dec 28 00:40:57.536990363 (537h11m57.408073398s) [noise=true , address=145.79.227.50:4373 , piece_id=3TI2DVFZABU4DGK74ONBWULNM45MNNQMSQVUWJP7WSY45XKN76UQ, seg_pos={0 0}, upload=471124]: Done uploading piece. err: <nil> Dec 28 00:40:57.540165880 (537h11m57.411248915s) [noise=true , address=145.79.227.15:4874 , piece_id=G4QDIAFCWDKAJYE4FXLSCGWXDC55CPYZJLZ5XYU6AQCT334BXZEQ, seg_pos={0 0}, upload=471124]: Done uploading piece. err: <nil> Dec 28 00:40:57.543176803 (537h11m57.414259838s) [noise=true , address=145.79.227.52:4446 , piece_id=6LJCHVETPYZYJ5CLXFFXQEDKYZTUEAFB7NTJPC2MI3PEKKUVEPFA, seg_pos={0 0}, upload=471124]: Done uploading piece. err: <nil> Dec 28 00:40:57.543532801 (537h11m57.414615856s) [noise=true , address=145.79.227.36:4517 , piece_id=35DVJKKSJOSLSPFNQHEM35GY6DWW7REUHISCJMZUQFILS2RKML2A, seg_pos={0 0}, upload=471124]: Done uploading piece. err: <nil> Dec 28 00:40:57.544639729 (537h11m57.415722764s) [noise=true , address=145.79.227.45:4230 , piece_id=5EQL36BUYUYYLJE5HCVNHS4YJ3IVZ5DEBWLBOGMMJ6T5IP2HTCSQ, seg_pos={0 0}, upload=471124]: Done uploading piece. err: <nil> Dec 28 00:40:57.548224996 (537h11m57.419308031s) [noise=true , address=145.79.227.11:4730 , piece_id=M2UQ5OLQJEBDUVZU3TVDPLQQ7MHZEDO4FGEVPFJACKQLBMA6EKOQ, seg_pos={0 0}, upload=471124]: Done uploading piece. err: <nil> Dec 28 00:40:57.550466864 (537h11m57.42154991s) [noise=true , address=145.79.227.38:4979 , piece_id=PWWIPBWNFMYYPDQBLCPZVYJAHUERO7LQGYGOY77MLSCUT2FFXHGQ, seg_pos={0 0}, upload=471124]: Done uploading piece. err: <nil> Dec 28 00:40:57.560546027 (537h11m57.431629072s) [noise=true , address=145.79.227.41:4087 , piece_id=ABG5YAGRUZ3DRBBYPU6LGJT6UWZLRXG4TC7RAO3ASCAU6VTZO3NQ, seg_pos={0 0}, upload=471124]: Done uploading piece. err: <nil> Dec 28 00:40:57.567171734 (537h11m57.438254779s) [noise=true , address=145.79.227.42:4123 , piece_id=CW26NMV2KRN2MUCYTSA3Z4VXMGHS25XLFHCOTBU2HXJJXIZCI75Q, seg_pos={0 0}, upload=471124]: Done uploading piece. err: <nil> Dec 28 00:40:57.574123311 (537h11m57.445206356s) [noise=true , address=145.79.227.16:4909 , piece_id=CUI5BXLCKAF6LYS5PVYD43SALWFUS4X7J5VIXXMBBPZK2H2EEBBQ, seg_pos={0 0}, upload=471124]: Done uploading piece. err: <nil> Dec 28 00:40:57.580757814 (537h11m57.451840849s) [noise=true , address=145.79.227.35:4481 , piece_id=S3AXSPCHPWF276YJ7WMDCJKHNPL5SA3PY37YUVTDGVDM2RDX76ZA, seg_pos={0 0}, upload=471124]: Done uploading piece. err: <nil> Dec 28 00:40:57.789048482 (537h11m57.660131517s) [noise=true , address=45.91.116.14:4190 , piece_id=7FFH3Q6MFSZAF42W6NBJ36P655RQML66A6MKHHBXCSKELC5LRPRA, seg_pos={0 0}, upload=471124]: Done uploading piece. err: <nil> Dec 28 00:40:57.791508427 (537h11m57.662591462s) [noise=true , address=45.91.116.11:4120 , piece_id=XP5TQTIW7GV5J6B2XOIGEDRCFAZ56FT2Z7IIN2E4RJUDZHRRAXLQ, seg_pos={0 0}, upload=471124]: Done uploading piece. err: <nil> Dec 28 00:40:57.791521481 (537h11m57.662604516s) [seg_pos={0 0}, upload=471124]: Segment reached optimal threshold of 13 pieces. Dec 28 00:40:57.791700844 (537h11m57.662783899s) [noise=true , address=145.79.227.7:4293 , piece_id=HEEHAFPHEG4FU4O43MW7WRSBMMQYMSJ2E6QGBYHIVH2CFTPLKIKQ, seg_pos={0 0}, upload=471124]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE): protocol: write tcp 188.166.225.170:52564->145.79.227.7:4293: use of closed network connection; write tcp 188.166.225.170:52564->145.79.227.7:4293: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52564->145.79.227.7:4293: use of closed network connection Dec 28 00:40:57.791709791 (537h11m57.662792836s) [noise=true , address=145.79.227.7:4293 , piece_id=HEEHAFPHEG4FU4O43MW7WRSBMMQYMSJ2E6QGBYHIVH2CFTPLKIKQ, seg_pos={0 0}, upload=471124]: Long tail cancellation due to optimal threshold Dec 28 00:40:57.791914260 (537h11m57.662997295s) [noise=true , address=45.91.116.7:4009 , piece_id=EPEVKHVQLF2BRZWX3HVPOO43YNABEHYAIXGJLBUFL6NDA7NM7BOA, seg_pos={0 0}, upload=471124]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:41098->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:41098->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41098->45.91.116.7:4009: use of closed network connection Dec 28 00:40:57.791921103 (537h11m57.663004138s) [noise=true , address=45.91.116.7:4009 , piece_id=EPEVKHVQLF2BRZWX3HVPOO43YNABEHYAIXGJLBUFL6NDA7NM7BOA, seg_pos={0 0}, upload=471124]: Long tail cancellation due to optimal threshold Dec 28 00:40:57.792030772 (537h11m57.663113807s) [noise=true , address=45.91.116.6:4976 , piece_id=YDY6JM7TPX2ODNQ6C4MAUY2H5XWLYIYRQZTXSEGM4EL2AQNOLXLQ, seg_pos={0 0}, upload=471124]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 00:40:57.792035861 (537h11m57.663118896s) [noise=true , address=45.91.116.6:4976 , piece_id=YDY6JM7TPX2ODNQ6C4MAUY2H5XWLYIYRQZTXSEGM4EL2AQNOLXLQ, seg_pos={0 0}, upload=471124]: Long tail cancellation due to optimal threshold Dec 28 00:40:57.792098851 (537h11m57.663181896s) [seg_pos={0 0}, upload=471124]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:40:57.792176509 (537h11m57.663259554s) [upload=471124]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:40:57.792181218 (537h11m57.663264253s) [upload=471124]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:40:58.089725471 (537h11m57.960808506s) [upload=471124]: Done uploading segments... Dec 28 00:40:58.089744628 (537h11m57.960827663s) [upload=471124]: Upload finished. err: <nil> Duration: 28.515124919s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 20.998318392s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 00:41:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 20.998347288s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [....X.X...X....X....X.XXX.X.X.XX.X.XXX...X.X.XX.XXX....X...X] duration: 30.004040283s trace: https://jaeger.dev.storj.io/trace/42c1eb9479fa933b FAIL (2025-12-28 00:41:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: 7FFH3Q6MFSZAF42W6NBJ36P655RQML66A6MKHHBXCSKELC5LRPRA) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: XP5TQTIW7GV5J6B2XOIGEDRCFAZ56FT2Z7IIN2E4RJUDZHRRAXLQ) context deadline exceeded ======LOGS====== gments... Dec 28 00:40:55.284980149 (537h11m55.156063184s) [upload=471124]: Got next segment. Inline: false Dec 28 00:40:55.285025015 (537h11m55.15610808s) [upload=471124]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:40:55.285030625 (537h11m55.15611367s) [upload=471124]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:40:55.531004529 (537h11m55.402087564s) [seg_pos={0 0}, upload=471124]: Begin upload segment... Dec 28 00:40:55.531052491 (537h11m55.402135526s) [seg_pos={0 0}, upload=471124]: Done begin upload segment. Dec 28 00:40:55.531079974 (537h11m55.402163009s) [noise=true , address=145.79.227.15:4874 , piece_id=G4QDIAFCWDKAJYE4FXLSCGWXDC55CPYZJLZ5XYU6AQCT334BXZEQ, seg_pos={0 0}, upload=471124]: Uploading piece... Dec 28 00:40:55.531185635 (537h11m55.40226868s) [noise=true , address=145.79.227.7:4293 , piece_id=HEEHAFPHEG4FU4O43MW7WRSBMMQYMSJ2E6QGBYHIVH2CFTPLKIKQ, seg_pos={0 0}, upload=471124]: Uploading piece... Dec 28 00:40:55.531955240 (537h11m55.403038275s) [noise=true , address=145.79.227.52:4446 , piece_id=6LJCHVETPYZYJ5CLXFFXQEDKYZTUEAFB7NTJPC2MI3PEKKUVEPFA, seg_pos={0 0}, upload=471124]: Uploading piece... Dec 28 00:40:55.532608002 (537h11m55.403691047s) [noise=true , address=45.91.116.6:4976 , piece_id=YDY6JM7TPX2ODNQ6C4MAUY2H5XWLYIYRQZTXSEGM4EL2AQNOLXLQ, seg_pos={0 0}, upload=471124]: Uploading piece... Dec 28 00:40:55.533163520 (537h11m55.404246555s) [noise=true , address=45.91.116.11:4120 , piece_id=XP5TQTIW7GV5J6B2XOIGEDRCFAZ56FT2Z7IIN2E4RJUDZHRRAXLQ, seg_pos={0 0}, upload=471124]: Uploading piece... Dec 28 00:40:55.533410761 (537h11m55.404493816s) [noise=true , address=145.79.227.11:4730 , piece_id=M2UQ5OLQJEBDUVZU3TVDPLQQ7MHZEDO4FGEVPFJACKQLBMA6EKOQ, seg_pos={0 0}, upload=471124]: Uploading piece... Dec 28 00:40:55.534007718 (537h11m55.405090754s) [noise=true , address=145.79.227.38:4979 , piece_id=PWWIPBWNFMYYPDQBLCPZVYJAHUERO7LQGYGOY77MLSCUT2FFXHGQ, seg_pos={0 0}, upload=471124]: Uploading piece... Dec 28 00:40:55.534125793 (537h11m55.405208828s) [noise=true , address=145.79.227.16:4909 , piece_id=CUI5BXLCKAF6LYS5PVYD43SALWFUS4X7J5VIXXMBBPZK2H2EEBBQ, seg_pos={0 0}, upload=471124]: Uploading piece... Dec 28 00:40:55.534435914 (537h11m55.405518949s) [noise=true , address=145.79.227.50:4373 , piece_id=3TI2DVFZABU4DGK74ONBWULNM45MNNQMSQVUWJP7WSY45XKN76UQ, seg_pos={0 0}, upload=471124]: Uploading piece... Dec 28 00:40:55.534760822 (537h11m55.405843857s) [noise=true , address=45.91.116.14:4190 , piece_id=7FFH3Q6MFSZAF42W6NBJ36P655RQML66A6MKHHBXCSKELC5LRPRA, seg_pos={0 0}, upload=471124]: Uploading piece... Dec 28 00:40:55.534837488 (537h11m55.405920523s) [noise=true , address=45.91.116.7:4009 , piece_id=EPEVKHVQLF2BRZWX3HVPOO43YNABEHYAIXGJLBUFL6NDA7NM7BOA, seg_pos={0 0}, upload=471124]: Uploading piece... Dec 28 00:40:55.535459903 (537h11m55.406542938s) [noise=true , address=145.79.227.42:4123 , piece_id=CW26NMV2KRN2MUCYTSA3Z4VXMGHS25XLFHCOTBU2HXJJXIZCI75Q, seg_pos={0 0}, upload=471124]: Uploading piece... Dec 28 00:40:55.536077189 (537h11m55.407160234s) [noise=true , address=145.79.227.45:4230 , piece_id=5EQL36BUYUYYLJE5HCVNHS4YJ3IVZ5DEBWLBOGMMJ6T5IP2HTCSQ, seg_pos={0 0}, upload=471124]: Uploading piece... Dec 28 00:40:55.536492960 (537h11m55.407576005s) [noise=true , address=145.79.227.35:4481 , piece_id=S3AXSPCHPWF276YJ7WMDCJKHNPL5SA3PY37YUVTDGVDM2RDX76ZA, seg_pos={0 0}, upload=471124]: Uploading piece... Dec 28 00:40:55.536878545 (537h11m55.40796159s) [noise=true , address=145.79.227.36:4517 , piece_id=35DVJKKSJOSLSPFNQHEM35GY6DWW7REUHISCJMZUQFILS2RKML2A, seg_pos={0 0}, upload=471124]: Uploading piece... Dec 28 00:40:55.541585487 (537h11m55.412668532s) [noise=true , address=145.79.227.41:4087 , piece_id=ABG5YAGRUZ3DRBBYPU6LGJT6UWZLRXG4TC7RAO3ASCAU6VTZO3NQ, seg_pos={0 0}, upload=471124]: Uploading piece... Dec 28 00:40:57.155294232 (537h11m57.026377277s) [upload=471124]: Next returned nil segment Dec 28 00:40:57.155313148 (537h11m57.026396203s) [upload=471124]: Waiting for error group managing segments... Dec 28 00:40:57.536990363 (537h11m57.408073398s) [noise=true , address=145.79.227.50:4373 , piece_id=3TI2DVFZABU4DGK74ONBWULNM45MNNQMSQVUWJP7WSY45XKN76UQ, seg_pos={0 0}, upload=471124]: Done uploading piece. err: <nil> Dec 28 00:40:57.540165880 (537h11m57.411248915s) [noise=true , address=145.79.227.15:4874 , piece_id=G4QDIAFCWDKAJYE4FXLSCGWXDC55CPYZJLZ5XYU6AQCT334BXZEQ, seg_pos={0 0}, upload=471124]: Done uploading piece. err: <nil> Dec 28 00:40:57.543176803 (537h11m57.414259838s) [noise=true , address=145.79.227.52:4446 , piece_id=6LJCHVETPYZYJ5CLXFFXQEDKYZTUEAFB7NTJPC2MI3PEKKUVEPFA, seg_pos={0 0}, upload=471124]: Done uploading piece. err: <nil> Dec 28 00:40:57.543532801 (537h11m57.414615856s) [noise=true , address=145.79.227.36:4517 , piece_id=35DVJKKSJOSLSPFNQHEM35GY6DWW7REUHISCJMZUQFILS2RKML2A, seg_pos={0 0}, upload=471124]: Done uploading piece. err: <nil> Dec 28 00:40:57.544639729 (537h11m57.415722764s) [noise=true , address=145.79.227.45:4230 , piece_id=5EQL36BUYUYYLJE5HCVNHS4YJ3IVZ5DEBWLBOGMMJ6T5IP2HTCSQ, seg_pos={0 0}, upload=471124]: Done uploading piece. err: <nil> Dec 28 00:40:57.548224996 (537h11m57.419308031s) [noise=true , address=145.79.227.11:4730 , piece_id=M2UQ5OLQJEBDUVZU3TVDPLQQ7MHZEDO4FGEVPFJACKQLBMA6EKOQ, seg_pos={0 0}, upload=471124]: Done uploading piece. err: <nil> Dec 28 00:40:57.550466864 (537h11m57.42154991s) [noise=true , address=145.79.227.38:4979 , piece_id=PWWIPBWNFMYYPDQBLCPZVYJAHUERO7LQGYGOY77MLSCUT2FFXHGQ, seg_pos={0 0}, upload=471124]: Done uploading piece. err: <nil> Dec 28 00:40:57.560546027 (537h11m57.431629072s) [noise=true , address=145.79.227.41:4087 , piece_id=ABG5YAGRUZ3DRBBYPU6LGJT6UWZLRXG4TC7RAO3ASCAU6VTZO3NQ, seg_pos={0 0}, upload=471124]: Done uploading piece. err: <nil> Dec 28 00:40:57.567171734 (537h11m57.438254779s) [noise=true , address=145.79.227.42:4123 , piece_id=CW26NMV2KRN2MUCYTSA3Z4VXMGHS25XLFHCOTBU2HXJJXIZCI75Q, seg_pos={0 0}, upload=471124]: Done uploading piece. err: <nil> Dec 28 00:40:57.574123311 (537h11m57.445206356s) [noise=true , address=145.79.227.16:4909 , piece_id=CUI5BXLCKAF6LYS5PVYD43SALWFUS4X7J5VIXXMBBPZK2H2EEBBQ, seg_pos={0 0}, upload=471124]: Done uploading piece. err: <nil> Dec 28 00:40:57.580757814 (537h11m57.451840849s) [noise=true , address=145.79.227.35:4481 , piece_id=S3AXSPCHPWF276YJ7WMDCJKHNPL5SA3PY37YUVTDGVDM2RDX76ZA, seg_pos={0 0}, upload=471124]: Done uploading piece. err: <nil> Dec 28 00:40:57.789048482 (537h11m57.660131517s) [noise=true , address=45.91.116.14:4190 , piece_id=7FFH3Q6MFSZAF42W6NBJ36P655RQML66A6MKHHBXCSKELC5LRPRA, seg_pos={0 0}, upload=471124]: Done uploading piece. err: <nil> Dec 28 00:40:57.791508427 (537h11m57.662591462s) [noise=true , address=45.91.116.11:4120 , piece_id=XP5TQTIW7GV5J6B2XOIGEDRCFAZ56FT2Z7IIN2E4RJUDZHRRAXLQ, seg_pos={0 0}, upload=471124]: Done uploading piece. err: <nil> Dec 28 00:40:57.791521481 (537h11m57.662604516s) [seg_pos={0 0}, upload=471124]: Segment reached optimal threshold of 13 pieces. Dec 28 00:40:57.791700844 (537h11m57.662783899s) [noise=true , address=145.79.227.7:4293 , piece_id=HEEHAFPHEG4FU4O43MW7WRSBMMQYMSJ2E6QGBYHIVH2CFTPLKIKQ, seg_pos={0 0}, upload=471124]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE): protocol: write tcp 188.166.225.170:52564->145.79.227.7:4293: use of closed network connection; write tcp 188.166.225.170:52564->145.79.227.7:4293: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52564->145.79.227.7:4293: use of closed network connection Dec 28 00:40:57.791709791 (537h11m57.662792836s) [noise=true , address=145.79.227.7:4293 , piece_id=HEEHAFPHEG4FU4O43MW7WRSBMMQYMSJ2E6QGBYHIVH2CFTPLKIKQ, seg_pos={0 0}, upload=471124]: Long tail cancellation due to optimal threshold Dec 28 00:40:57.791914260 (537h11m57.662997295s) [noise=true , address=45.91.116.7:4009 , piece_id=EPEVKHVQLF2BRZWX3HVPOO43YNABEHYAIXGJLBUFL6NDA7NM7BOA, seg_pos={0 0}, upload=471124]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:41098->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:41098->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41098->45.91.116.7:4009: use of closed network connection Dec 28 00:40:57.791921103 (537h11m57.663004138s) [noise=true , address=45.91.116.7:4009 , piece_id=EPEVKHVQLF2BRZWX3HVPOO43YNABEHYAIXGJLBUFL6NDA7NM7BOA, seg_pos={0 0}, upload=471124]: Long tail cancellation due to optimal threshold Dec 28 00:40:57.792030772 (537h11m57.663113807s) [noise=true , address=45.91.116.6:4976 , piece_id=YDY6JM7TPX2ODNQ6C4MAUY2H5XWLYIYRQZTXSEGM4EL2AQNOLXLQ, seg_pos={0 0}, upload=471124]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 00:40:57.792035861 (537h11m57.663118896s) [noise=true , address=45.91.116.6:4976 , piece_id=YDY6JM7TPX2ODNQ6C4MAUY2H5XWLYIYRQZTXSEGM4EL2AQNOLXLQ, seg_pos={0 0}, upload=471124]: Long tail cancellation due to optimal threshold Dec 28 00:40:57.792098851 (537h11m57.663181896s) [seg_pos={0 0}, upload=471124]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:40:57.792176509 (537h11m57.663259554s) [upload=471124]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:40:57.792181218 (537h11m57.663264253s) [upload=471124]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:40:58.089725471 (537h11m57.960808506s) [upload=471124]: Done uploading segments... Dec 28 00:40:58.089744628 (537h11m57.960827663s) [upload=471124]: Upload finished. err: <nil> Duration: 30.004063158s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 42.456522716s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 00:41:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 42.456548896s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 25.436666826s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 00:40:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 25.436703825s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [X....X.X...X....X....X.XXX.X.X.XX.X.XXX...X.X.XX.XXX....X...] duration: 5.984142791s trace: https://jaeger.dev.storj.io/trace/6d419d1788612851 FAIL (2025-12-28 00:40:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 00:38:53.736041728 (537h9m53.607124763s) [upload=471094]: Starting upload Dec 28 00:38:53.736100290 (537h9m53.607183335s) [upload=471094]: Done starting upload Dec 28 00:38:53.736214197 (537h9m53.607297232s) [upload=471094]: Uploading segments... Dec 28 00:38:53.736308176 (537h9m53.607391211s) [upload=471094]: Got next segment. Inline: false Dec 28 00:38:53.736345607 (537h9m53.607428642s) [upload=471094]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 00:38:53.736350376 (537h9m53.607433421s) [upload=471094]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 00:38:53.978532388 (537h9m53.849615423s) [seg_pos={0 0}, upload=471094]: Begin upload segment... Dec 28 00:38:53.978618543 (537h9m53.849701578s) [seg_pos={0 0}, upload=471094]: Done begin upload segment. Dec 28 00:38:53.978660833 (537h9m53.849743878s) [noise=true , address=77.90.31.21:4878 , piece_id=7DT5V5KIQDYTEC56Q2C4LRGJJ2ZQSDSIN3UUDUDKMSGLH7VOMTVA, seg_pos={0 0}, upload=471094]: Uploading piece... Dec 28 00:38:53.978712181 (537h9m53.849795236s) [noise=true , address=77.90.31.13:4627 , piece_id=RBVWKFAWZHKZCBOPENRMGALLHWN6HFLLIDO3N2WGLNGG5HQPCNPA, seg_pos={0 0}, upload=471094]: Uploading piece... Dec 28 00:38:53.979258651 (537h9m53.850341686s) [noise=true , address=145.79.227.42:4122 , piece_id=TWEABJMFVY2P72WL4UOHQUNRK22TI7BMJUMWELOYIAFNKEUOXJBQ, seg_pos={0 0}, upload=471094]: Uploading piece... Dec 28 00:38:53.979572639 (537h9m53.850655674s) [noise=true , address=77.90.31.15:4697 , piece_id=E4WAE2D4JVCK4E3XNY3OYKOSP3ETWBJU36LCRBDQPF3AIJQYELFQ, seg_pos={0 0}, upload=471094]: Uploading piece... Dec 28 00:38:53.979853484 (537h9m53.850936519s) [noise=true , address=145.79.227.37:4566 , piece_id=ZQAE4OLGYHRDENELMN4XKDAJJBLLSPR5SR3CKGQ5TAG7IS2PQ2YA, seg_pos={0 0}, upload=471094]: Uploading piece... Dec 28 00:38:53.980129790 (537h9m53.851212835s) [noise=true , address=145.79.227.45:4230 , piece_id=TXSB36OQ365XXRBUZOVOGRCCOKQIOXWV2M2TUF4BK5MV35JYKUMA, seg_pos={0 0}, upload=471094]: Uploading piece... Dec 28 00:38:53.980398211 (537h9m53.851481246s) [noise=true , address=145.79.227.34:4907 , piece_id=N2TMRN7JMVLA4T3G34FIEVR4CNDPA7XVQX66WNAIXJWOB7TJKFSA, seg_pos={0 0}, upload=471094]: Uploading piece... Dec 28 00:38:53.980675530 (537h9m53.851758575s) [noise=true , address=145.79.227.11:4731 , piece_id=KCWUIDQYZJFLKUQG37YXIAKC2DVDZZEMREQFBXMC4MWLORKCI3MA, seg_pos={0 0}, upload=471094]: Uploading piece... Dec 28 00:38:53.980997713 (537h9m53.852080748s) [noise=true , address=77.90.31.19:4842 , piece_id=R2YVKDMSGYBBHZLTJHPMAL4IXICXPPGEHOJPXU35KGZRQOB7BMHA, seg_pos={0 0}, upload=471094]: Uploading piece... Dec 28 00:38:53.981375432 (537h9m53.852458467s) [noise=true , address=145.79.227.50:4374 , piece_id=QQTW2GMYQGGHQS5FXLWATPFS3NLMUBAAX3GVXYZOGNQWFQEVEKCQ, seg_pos={0 0}, upload=471094]: Uploading piece... Dec 28 00:38:53.982560278 (537h9m53.853643313s) [noise=true , address=145.79.227.48:4302 , piece_id=4W5GYPYITBKUB6LDWKYSIUTSWY5ARP4TNGOGHQMP7SQB4XBHUQBQ, seg_pos={0 0}, upload=471094]: Uploading piece... Dec 28 00:38:53.984086505 (537h9m53.85516954s) [noise=true , address=145.79.227.13:4804 , piece_id=KBDGAUZ6PI45G435CSGSOFREMQ36MI3HMW6CMSD5ZHFECTLDNBTQ, seg_pos={0 0}, upload=471094]: Uploading piece... Dec 28 00:38:53.986088206 (537h9m53.857171241s) [noise=true , address=145.79.227.36:4519 , piece_id=SLMWI5GS7ZPRBKO7SETHXAP7V6K5Z5V4ISWRIIMIRO5ON27LEZDA, seg_pos={0 0}, upload=471094]: Uploading piece... Dec 28 00:38:53.987388613 (537h9m53.858471648s) [noise=true , address=145.79.227.38:4979 , piece_id=KH3GSBPUL52V3IUSZYOHWBNBMR4SWANFJJ7EQCMXJIPFNEZJQWTQ, seg_pos={0 0}, upload=471094]: Uploading piece... Dec 28 00:38:53.988469230 (537h9m53.859552265s) [noise=true , address=45.91.116.7:4009 , piece_id=IZTKLOQLLOJX2F36ZKSSB5XWE7U6BHYACB75HEIO3MP3H7WWY6CA, seg_pos={0 0}, upload=471094]: Uploading piece... Dec 28 00:38:53.989531422 (537h9m53.860614457s) [noise=true , address=45.91.116.4:4939 , piece_id=6VWG77X5E2VYBJIMFKSZTEG5S6Q25YPHKVFKXHEQUFPUHPGB2MAQ, seg_pos={0 0}, upload=471094]: Uploading piece... Dec 28 00:38:55.656510992 (537h9m55.527594027s) [upload=471094]: Next returned nil segment Dec 28 00:38:55.656530158 (537h9m55.527613193s) [upload=471094]: Waiting for error group managing segments... Dec 28 00:38:55.977150963 (537h9m55.848233998s) [noise=true , address=145.79.227.11:4731 , piece_id=KCWUIDQYZJFLKUQG37YXIAKC2DVDZZEMREQFBXMC4MWLORKCI3MA, seg_pos={0 0}, upload=471094]: Done uploading piece. err: <nil> Dec 28 00:38:55.981769397 (537h9m55.852852432s) [noise=true , address=145.79.227.13:4804 , piece_id=KBDGAUZ6PI45G435CSGSOFREMQ36MI3HMW6CMSD5ZHFECTLDNBTQ, seg_pos={0 0}, upload=471094]: Done uploading piece. err: <nil> Dec 28 00:38:55.987409537 (537h9m55.858492582s) [noise=true , address=145.79.227.36:4519 , piece_id=SLMWI5GS7ZPRBKO7SETHXAP7V6K5Z5V4ISWRIIMIRO5ON27LEZDA, seg_pos={0 0}, upload=471094]: Done uploading piece. err: <nil> Dec 28 00:38:55.991200095 (537h9m55.86228313s) [noise=true , address=145.79.227.42:4122 , piece_id=TWEABJMFVY2P72WL4UOHQUNRK22TI7BMJUMWELOYIAFNKEUOXJBQ, seg_pos={0 0}, upload=471094]: Done uploading piece. err: <nil> Dec 28 00:38:56.001430455 (537h9m55.87251351s) [noise=true , address=145.79.227.45:4230 , piece_id=TXSB36OQ365XXRBUZOVOGRCCOKQIOXWV2M2TUF4BK5MV35JYKUMA, seg_pos={0 0}, upload=471094]: Done uploading piece. err: <nil> Dec 28 00:38:56.017887102 (537h9m55.888970167s) [noise=true , address=145.79.227.38:4979 , piece_id=KH3GSBPUL52V3IUSZYOHWBNBMR4SWANFJJ7EQCMXJIPFNEZJQWTQ, seg_pos={0 0}, upload=471094]: Done uploading piece. err: <nil> Dec 28 00:38:56.027593744 (537h9m55.898676789s) [noise=true , address=145.79.227.37:4566 , piece_id=ZQAE4OLGYHRDENELMN4XKDAJJBLLSPR5SR3CKGQ5TAG7IS2PQ2YA, seg_pos={0 0}, upload=471094]: Done uploading piece. err: <nil> Dec 28 00:38:56.139443043 (537h9m56.010526098s) [noise=true , address=145.79.227.50:4374 , piece_id=QQTW2GMYQGGHQS5FXLWATPFS3NLMUBAAX3GVXYZOGNQWFQEVEKCQ, seg_pos={0 0}, upload=471094]: Done uploading piece. err: <nil> Dec 28 00:38:56.258130162 (537h9m56.129213208s) [noise=true , address=77.90.31.13:4627 , piece_id=RBVWKFAWZHKZCBOPENRMGALLHWN6HFLLIDO3N2WGLNGG5HQPCNPA, seg_pos={0 0}, upload=471094]: Done uploading piece. err: <nil> Dec 28 00:38:56.318612551 (537h9m56.189695586s) [noise=true , address=45.91.116.4:4939 , piece_id=6VWG77X5E2VYBJIMFKSZTEG5S6Q25YPHKVFKXHEQUFPUHPGB2MAQ, seg_pos={0 0}, upload=471094]: Done uploading piece. err: <nil> Dec 28 00:38:56.320126954 (537h9m56.191209989s) [noise=true , address=77.90.31.15:4697 , piece_id=E4WAE2D4JVCK4E3XNY3OYKOSP3ETWBJU36LCRBDQPF3AIJQYELFQ, seg_pos={0 0}, upload=471094]: Done uploading piece. err: <nil> Dec 28 00:38:56.841900666 (537h9m56.712983712s) [noise=true , address=77.90.31.19:4842 , piece_id=R2YVKDMSGYBBHZLTJHPMAL4IXICXPPGEHOJPXU35KGZRQOB7BMHA, seg_pos={0 0}, upload=471094]: Done uploading piece. err: <nil> Dec 28 00:38:57.079215101 (537h9m56.950298136s) [noise=true , address=77.90.31.21:4878 , piece_id=7DT5V5KIQDYTEC56Q2C4LRGJJ2ZQSDSIN3UUDUDKMSGLH7VOMTVA, seg_pos={0 0}, upload=471094]: Done uploading piece. err: <nil> Dec 28 00:38:57.079231932 (537h9m56.950314967s) [seg_pos={0 0}, upload=471094]: Segment reached optimal threshold of 13 pieces. Dec 28 00:38:57.079390014 (537h9m56.950473049s) [noise=true , address=45.91.116.7:4009 , piece_id=IZTKLOQLLOJX2F36ZKSSB5XWE7U6BHYACB75HEIO3MP3H7WWY6CA, seg_pos={0 0}, upload=471094]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 28 00:38:57.079396506 (537h9m56.950479541s) [noise=true , address=45.91.116.7:4009 , piece_id=IZTKLOQLLOJX2F36ZKSSB5XWE7U6BHYACB75HEIO3MP3H7WWY6CA, seg_pos={0 0}, upload=471094]: Long tail cancellation due to optimal threshold Dec 28 00:38:57.079507467 (537h9m56.950590522s) [noise=true , address=145.79.227.34:4907 , piece_id=N2TMRN7JMVLA4T3G34FIEVR4CNDPA7XVQX66WNAIXJWOB7TJKFSA, seg_pos={0 0}, upload=471094]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 28 00:38:57.079517387 (537h9m56.950600432s) [noise=true , address=145.79.227.34:4907 , piece_id=N2TMRN7JMVLA4T3G34FIEVR4CNDPA7XVQX66WNAIXJWOB7TJKFSA, seg_pos={0 0}, upload=471094]: Long tail cancellation due to optimal threshold Dec 28 00:38:57.079619121 (537h9m56.950702166s) [noise=true , address=145.79.227.48:4302 , piece_id=4W5GYPYITBKUB6LDWKYSIUTSWY5ARP4TNGOGHQMP7SQB4XBHUQBQ, seg_pos={0 0}, upload=471094]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tGpuYdPLiy4ncDjxrZbQfp7Bq79NVBHDmr4vjaL4qMpudrvgR): protocol: write tcp 188.166.225.170:58556->145.79.227.48:4302: use of closed network connection; write tcp 188.166.225.170:58556->145.79.227.48:4302: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58556->145.79.227.48:4302: use of closed network connection Dec 28 00:38:57.079628949 (537h9m56.950712014s) [noise=true , address=145.79.227.48:4302 , piece_id=4W5GYPYITBKUB6LDWKYSIUTSWY5ARP4TNGOGHQMP7SQB4XBHUQBQ, seg_pos={0 0}, upload=471094]: Long tail cancellation due to optimal threshold Dec 28 00:38:57.079652153 (537h9m56.950735219s) [seg_pos={0 0}, upload=471094]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 00:38:57.079743707 (537h9m56.950826772s) [upload=471094]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:38:57.079750159 (537h9m56.950833204s) [upload=471094]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:38:57.365408852 (537h9m57.236491887s) [upload=471094]: Done uploading segments... Dec 28 00:38:57.365427627 (537h9m57.236510662s) [upload=471094]: Upload finished. err: <nil> Duration: 5.984172857s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 27.710772411s trace: https://jaeger.dev.storj.io/trace/3df1366b4347b6d4 FAIL (2025-12-28 00:40:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 00:20:07.252792744 (536h51m7.123875779s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253129255 (536h51m7.12421229s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253449785 (536h51m7.12453282s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.253736862 (536h51m7.124819897s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254078042 (536h51m7.125161077s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.254536344 (536h51m7.125619389s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.256351180 (536h51m7.127434225s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.257369059 (536h51m7.128452104s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.259243357 (536h51m7.130326402s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.261241703 (536h51m7.132324738s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.262967628 (536h51m7.134050684s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.264704265 (536h51m7.13578731s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.266073412 (536h51m7.137156457s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267036707 (536h51m7.138119752s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.267694630 (536h51m7.138777675s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:07.268804132 (536h51m7.139887178s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Uploading piece... Dec 28 00:20:08.626285899 (536h51m8.497368964s) [upload=470822]: Next returned nil segment Dec 28 00:20:08.626301989 (536h51m8.497385034s) [upload=470822]: Waiting for error group managing segments... Dec 28 00:20:08.975106801 (536h51m8.846189836s) [noise=true , address=64.49.13.154:4926 , piece_id=6FKFSPAUR2QLZJ73CBP5XTFCXMFEQBL3CSEMJMZMFWBM7JIKVL3Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:08.983435119 (536h51m8.854518154s) [noise=true , address=64.49.13.155:4920 , piece_id=WIPG52BQDDBDKRKMZVA5WMBY5JHCZ7M5LFQXH2PL3L72MBS2Y26A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.147476892 (536h51m9.018559937s) [noise=true , address=145.79.227.20:4446 , piece_id=NZEC55HJD6B4HQZWDVEJGVGFDSCR4W44WMBLZJNEEOJXRIRIUO7A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.153097544 (536h51m9.024180589s) [noise=true , address=145.79.227.18:4015 , piece_id=I7ASE4557VVONV2S4E7NFNN44S3AJ7HSSL7V4SA37NPMDDDXHV2A, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.286268533 (536h51m9.157351578s) [noise=true , address=77.90.31.21:4888 , piece_id=3CCDNBMESKEYK2NO2RG44QVXYG4Z66PRJLKAUB36WJAAJ77524NA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.318954332 (536h51m9.190037377s) [noise=true , address=77.90.31.14:4692 , piece_id=ZO7XDGVBYKP3ZNZFAWUKDLBZDFFUDYN4CIHG5V3QWQ26QXZ62UDA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.344638429 (536h51m9.215721464s) [noise=true , address=77.90.31.15:4712 , piece_id=WHVLK7EF3VMNAOOGOD5B3VBBL4ZD3EG7X62UAUOKQNWHZYMWKWEA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.351873285 (536h51m9.22295632s) [noise=true , address=64.49.13.26:4948 , piece_id=BMODLXT7ZEZXQZF4D6RC2F2MYIQ6DUM22O2AOAUP5Q7ZKKHKQGCA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:09.398037139 (536h51m9.269120184s) [noise=true , address=77.90.31.19:4858 , piece_id=ZVDXEQ2UKWB5GT3SJFT3CMUTWYJRBHUMMWGZ2GV6TFNDQV22U43Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.000052464 (536h51m9.87113552s) [noise=true , address=145.79.227.30:4769 , piece_id=VWK2ZMI25IWI3CYUEZQZVJAOA3DGBE7YQW7IKPQXBRQXGTML5RWA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.026473574 (536h51m9.897556609s) [noise=true , address=145.79.227.44:4218 , piece_id=K7GBYGXPOCK6SSZVPIKWVVZCDA2TDSSZVLOCXOHBH4VVIJ26XCDQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.029899798 (536h51m9.900982833s) [noise=true , address=145.79.227.44:4228 , piece_id=E6HQBBHWQBR6EKRMK2QX5QGHEBWL3UVST6ZWHJZSBLQCGR5HC2UQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.060053706 (536h51m9.931136751s) [noise=true , address=145.79.227.30:4782 , piece_id=VYE2MH46IJUS5DX3WCXYKFCP47GLAWOR2WPX7UPYBXES3HPITBUQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091223749 (536h51m9.962306784s) [noise=true , address=145.79.227.43:4180 , piece_id=SBDBKHAAPB3N2OB5SHCRWEK2QLISJH3XDTYSXNHU6PG4MUPLRI4Q, seg_pos={0 0}, upload=470822]: Done uploading piece. err: <nil> Dec 28 00:20:10.091243617 (536h51m9.962326662s) [seg_pos={0 0}, upload=470822]: Segment reached optimal threshold of 14 pieces. Dec 28 00:20:10.091446984 (536h51m9.962530029s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091453597 (536h51m9.962536632s) [noise=true , address=145.79.227.40:4066 , piece_id=SKUJ43IHW7RZY3ZWRF6PJNVW2JUEPSXLSRYBOHYCKSZUSY56OQVQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091486379 (536h51m9.962569454s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46304->64.49.13.169:4709: use of closed network connection Dec 28 00:20:10.091505225 (536h51m9.96258827s) [noise=true , address=64.49.13.169:4709 , piece_id=6BJ7YXX53YSRRPCKE5SPHOLY5AIX5VR2NCAIJEWCU6B7AVLJ4WHQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091572803 (536h51m9.962655838s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091577923 (536h51m9.962660958s) [noise=true , address=145.79.227.33:4891 , piece_id=KWRKDPTQR5UQ2YCJHL4UFPKOE5PKXS5ANV5ZMW7FCJ6EYU74LGQQ, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091667614 (536h51m9.962750649s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14Pqp5cbhmd44QnikAuD5TgsKFDR3gB2qtLuVSroaWVv4U2cDh): protocol: expected piece hash; context canceled; EOF Dec 28 00:20:10.091672654 (536h51m9.962755699s) [noise=true , address=145.79.227.34:4909 , piece_id=GCPU3OYH7VRHXKLO2KUM66GQOKCGI5KOSOWOKTMZ2B3X3OAZOGLA, seg_pos={0 0}, upload=470822]: Long tail cancellation due to optimal threshold Dec 28 00:20:10.091722408 (536h51m9.962805453s) [seg_pos={0 0}, upload=470822]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 00:20:10.091784967 (536h51m9.962868002s) [upload=470822]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 00:20:10.091790197 (536h51m9.962873242s) [upload=470822]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 00:20:10.383220468 (536h51m10.254303503s) [upload=470822]: Done uploading segments... Dec 28 00:20:10.383244895 (536h51m10.25432794s) [upload=470822]: Upload finished. err: <nil> Duration: 27.710799924s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)