Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [...X..XX.....XXXX.....XXX....XX....X..X......X.X..XX.....X..] duration: 5.98706514s trace: https://jaeger.dev.storj.io/trace/25f0a86b327adac5 FAIL (2025-12-26 12:54:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 4AQ7JCS2YYX3BUINUIE375WQFUU6VNBADRTYAHITMK724H5QUARQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: FNMHETMVEEU7YVY5Z527MK6CTPI5TSXYTDIRP5OMFS3YSG47EEHQ) context deadline exceeded ======LOGS====== segments... Dec 26 12:53:33.060139405 (501h24m32.93122247s) [upload=440045]: Got next segment. Inline: false Dec 26 12:53:33.060183419 (501h24m32.931266454s) [upload=440045]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:53:33.060189249 (501h24m32.931272295s) [upload=440045]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:53:33.304580344 (501h24m33.175663389s) [seg_pos={0 0}, upload=440045]: Begin upload segment... Dec 26 12:53:33.304632342 (501h24m33.175715377s) [seg_pos={0 0}, upload=440045]: Done begin upload segment. Dec 26 12:53:33.304685023 (501h24m33.175768068s) [noise=true , address=145.79.227.16:4910 , piece_id=VMMXT72NDQ4QZG6ELIFOCX4DGTYDLU4YMRWJTZAA46VDTMEAG6YA, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.304695913 (501h24m33.175778959s) [noise=true , address=45.91.116.13:4154 , piece_id=4AQ7JCS2YYX3BUINUIE375WQFUU6VNBADRTYAHITMK724H5QUARQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.304862942 (501h24m33.175945977s) [noise=true , address=45.91.116.14:4189 , piece_id=FNMHETMVEEU7YVY5Z527MK6CTPI5TSXYTDIRP5OMFS3YSG47EEHQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.305344799 (501h24m33.176427844s) [noise=true , address=145.79.227.4:4185 , piece_id=5FEZC7ARULDUTJGMZHLHO7NBQBV6NJJWIRHR74XKUJDD544W47OA, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.305502649 (501h24m33.176585685s) [noise=true , address=145.79.227.46:4602 , piece_id=734WH2AFE2ZXM7OEWH55N2YCJGKYA7QJC2UAEKG7S27K7O4AAICQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.305928852 (501h24m33.177011887s) [noise=true , address=77.90.31.14:4664 , piece_id=DEUYUFUOQ5ZJ64MC47P33XFP5WSDFQ35LUAG7JUOLRR64XAQLGMQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.310305796 (501h24m33.181388851s) [noise=true , address=145.79.227.52:4446 , piece_id=GE63KAOXOW5OQOJFQKD2OCRW2NHRWQ7ULBEVNW5OCVCVQFQ5N72Q, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.310942769 (501h24m33.182025814s) [noise=true , address=45.91.116.6:4976 , piece_id=Z64GC2HQ6FXCJKX7ITZTWRIUNMAYGOIYHAXGTK42P6PJMG235DPQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.311429736 (501h24m33.182512781s) [noise=true , address=145.79.227.35:4481 , piece_id=KPKOFEE7P64X6K64CATZCX2PWSGQA4VVMC2TFQTBSSO7JZNF3QKA, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.311996105 (501h24m33.18307915s) [noise=true , address=145.79.227.45:4230 , piece_id=7NZXREFL4LBCBY6JZVGJSJFS2NYKVPSIQ4PE727BDANLV5ZSPGYQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.312483022 (501h24m33.183566067s) [noise=true , address=145.79.227.49:4337 , piece_id=YXIJMAZVJIEVFIXLRKMG3Z3BT2UCSGM2KUAUNTYWVMTDGKPD6K2Q, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.313996513 (501h24m33.185079558s) [noise=true , address=145.79.227.41:4086 , piece_id=3KQWI5WAPGUNIHIR24NKYKXQ2IDTBX4G6ZZX3UKHZSYHZKFSUAOQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.315528761 (501h24m33.186611806s) [noise=true , address=145.79.227.39:4013 , piece_id=BBAUK5WO4SYS7ZQYZWVYIVLYONF2VLIYAX3Q7PXZBXAU2K5FGULQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.316907637 (501h24m33.187990682s) [noise=true , address=145.79.227.37:4567 , piece_id=LTGY5WRV5WFRAHFW2CT5EQZPRDXLUVCW7XMKMCIRHQIZXOG3BZ7Q, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.319905165 (501h24m33.19098822s) [noise=true , address=45.91.116.11:4117 , piece_id=XGOCYUWEFN5FKC2XLO22VM2EYOMQL5ELL6ZJMUFPBRETUYMF443Q, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.321664966 (501h24m33.192748001s) [noise=true , address=145.79.227.38:4979 , piece_id=3KSMBSFYPFHKW55E4LWN6AUH64NPSYKI5ZXOTKM4ZG4XOYQHAVFQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:34.914820462 (501h24m34.785903507s) [upload=440045]: Next returned nil segment Dec 26 12:53:34.914837635 (501h24m34.78592069s) [upload=440045]: Waiting for error group managing segments... Dec 26 12:53:35.298879578 (501h24m35.169962623s) [noise=true , address=145.79.227.46:4602 , piece_id=734WH2AFE2ZXM7OEWH55N2YCJGKYA7QJC2UAEKG7S27K7O4AAICQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.308194576 (501h24m35.179277611s) [noise=true , address=145.79.227.4:4185 , piece_id=5FEZC7ARULDUTJGMZHLHO7NBQBV6NJJWIRHR74XKUJDD544W47OA, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.321282087 (501h24m35.192365122s) [noise=true , address=145.79.227.49:4337 , piece_id=YXIJMAZVJIEVFIXLRKMG3Z3BT2UCSGM2KUAUNTYWVMTDGKPD6K2Q, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.324654759 (501h24m35.195737804s) [noise=true , address=145.79.227.37:4567 , piece_id=LTGY5WRV5WFRAHFW2CT5EQZPRDXLUVCW7XMKMCIRHQIZXOG3BZ7Q, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.328057719 (501h24m35.199140754s) [noise=true , address=145.79.227.45:4230 , piece_id=7NZXREFL4LBCBY6JZVGJSJFS2NYKVPSIQ4PE727BDANLV5ZSPGYQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.336066038 (501h24m35.207149073s) [noise=true , address=145.79.227.38:4979 , piece_id=3KSMBSFYPFHKW55E4LWN6AUH64NPSYKI5ZXOTKM4ZG4XOYQHAVFQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.336911889 (501h24m35.207994934s) [noise=true , address=145.79.227.16:4910 , piece_id=VMMXT72NDQ4QZG6ELIFOCX4DGTYDLU4YMRWJTZAA46VDTMEAG6YA, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.344439162 (501h24m35.215522227s) [noise=true , address=145.79.227.41:4086 , piece_id=3KQWI5WAPGUNIHIR24NKYKXQ2IDTBX4G6ZZX3UKHZSYHZKFSUAOQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.344909568 (501h24m35.215992603s) [noise=true , address=145.79.227.39:4013 , piece_id=BBAUK5WO4SYS7ZQYZWVYIVLYONF2VLIYAX3Q7PXZBXAU2K5FGULQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.351561845 (501h24m35.22264488s) [noise=true , address=145.79.227.35:4481 , piece_id=KPKOFEE7P64X6K64CATZCX2PWSGQA4VVMC2TFQTBSSO7JZNF3QKA, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.352409338 (501h24m35.223492383s) [noise=true , address=145.79.227.52:4446 , piece_id=GE63KAOXOW5OQOJFQKD2OCRW2NHRWQ7ULBEVNW5OCVCVQFQ5N72Q, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.601492897 (501h24m35.472575942s) [noise=true , address=45.91.116.14:4189 , piece_id=FNMHETMVEEU7YVY5Z527MK6CTPI5TSXYTDIRP5OMFS3YSG47EEHQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.610741338 (501h24m35.481824383s) [noise=true , address=45.91.116.13:4154 , piece_id=4AQ7JCS2YYX3BUINUIE375WQFUU6VNBADRTYAHITMK724H5QUARQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.610765683 (501h24m35.481848729s) [seg_pos={0 0}, upload=440045]: Segment reached optimal threshold of 13 pieces. Dec 26 12:53:35.610962629 (501h24m35.482045674s) [noise=true , address=45.91.116.11:4117 , piece_id=XGOCYUWEFN5FKC2XLO22VM2EYOMQL5ELL6ZJMUFPBRETUYMF443Q, seg_pos={0 0}, upload=440045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 26 12:53:35.610968921 (501h24m35.482051956s) [noise=true , address=45.91.116.11:4117 , piece_id=XGOCYUWEFN5FKC2XLO22VM2EYOMQL5ELL6ZJMUFPBRETUYMF443Q, seg_pos={0 0}, upload=440045]: Long tail cancellation due to optimal threshold Dec 26 12:53:35.611078359 (501h24m35.482161394s) [noise=true , address=77.90.31.14:4664 , piece_id=DEUYUFUOQ5ZJ64MC47P33XFP5WSDFQ35LUAG7JUOLRR64XAQLGMQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: write tcp 188.166.225.170:54322->77.90.31.14:4664: use of closed network connection; write tcp 188.166.225.170:54322->77.90.31.14:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54322->77.90.31.14:4664: use of closed network connection Dec 26 12:53:35.611088449 (501h24m35.482171494s) [noise=true , address=77.90.31.14:4664 , piece_id=DEUYUFUOQ5ZJ64MC47P33XFP5WSDFQ35LUAG7JUOLRR64XAQLGMQ, seg_pos={0 0}, upload=440045]: Long tail cancellation due to optimal threshold Dec 26 12:53:35.611146599 (501h24m35.482229634s) [noise=true , address=45.91.116.6:4976 , piece_id=Z64GC2HQ6FXCJKX7ITZTWRIUNMAYGOIYHAXGTK42P6PJMG235DPQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:53846->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:53846->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53846->45.91.116.6:4976: use of closed network connection Dec 26 12:53:35.611152770 (501h24m35.482235816s) [noise=true , address=45.91.116.6:4976 , piece_id=Z64GC2HQ6FXCJKX7ITZTWRIUNMAYGOIYHAXGTK42P6PJMG235DPQ, seg_pos={0 0}, upload=440045]: Long tail cancellation due to optimal threshold Dec 26 12:53:35.611193359 (501h24m35.482276404s) [seg_pos={0 0}, upload=440045]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:53:35.611282528 (501h24m35.482365583s) [upload=440045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:53:35.611289651 (501h24m35.482372686s) [upload=440045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:53:35.896282848 (501h24m35.767365894s) [upload=440045]: Done uploading segments... Dec 26 12:53:35.896302896 (501h24m35.767385931s) [upload=440045]: Upload finished. err: <nil> Duration: 5.987091531s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [...X.......XXX...X.....X.......X.XX......XXX..XXXXXXX.XXXX.X] duration: 5.001458211s trace: https://jaeger.dev.storj.io/trace/3060f5237e3dbb77 FAIL (2025-12-26 12:54:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001480875s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X...X..XX.....XXXX.....XXX....XX....X..X......X.X..XX.....X.] duration: 5.898239159s trace: https://jaeger.dev.storj.io/trace/25f0a86b327adac5 FAIL (2025-12-26 12:54:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 4AQ7JCS2YYX3BUINUIE375WQFUU6VNBADRTYAHITMK724H5QUARQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: FNMHETMVEEU7YVY5Z527MK6CTPI5TSXYTDIRP5OMFS3YSG47EEHQ) context deadline exceeded ======LOGS====== segments... Dec 26 12:53:33.060139405 (501h24m32.93122247s) [upload=440045]: Got next segment. Inline: false Dec 26 12:53:33.060183419 (501h24m32.931266454s) [upload=440045]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:53:33.060189249 (501h24m32.931272295s) [upload=440045]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:53:33.304580344 (501h24m33.175663389s) [seg_pos={0 0}, upload=440045]: Begin upload segment... Dec 26 12:53:33.304632342 (501h24m33.175715377s) [seg_pos={0 0}, upload=440045]: Done begin upload segment. Dec 26 12:53:33.304685023 (501h24m33.175768068s) [noise=true , address=145.79.227.16:4910 , piece_id=VMMXT72NDQ4QZG6ELIFOCX4DGTYDLU4YMRWJTZAA46VDTMEAG6YA, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.304695913 (501h24m33.175778959s) [noise=true , address=45.91.116.13:4154 , piece_id=4AQ7JCS2YYX3BUINUIE375WQFUU6VNBADRTYAHITMK724H5QUARQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.304862942 (501h24m33.175945977s) [noise=true , address=45.91.116.14:4189 , piece_id=FNMHETMVEEU7YVY5Z527MK6CTPI5TSXYTDIRP5OMFS3YSG47EEHQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.305344799 (501h24m33.176427844s) [noise=true , address=145.79.227.4:4185 , piece_id=5FEZC7ARULDUTJGMZHLHO7NBQBV6NJJWIRHR74XKUJDD544W47OA, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.305502649 (501h24m33.176585685s) [noise=true , address=145.79.227.46:4602 , piece_id=734WH2AFE2ZXM7OEWH55N2YCJGKYA7QJC2UAEKG7S27K7O4AAICQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.305928852 (501h24m33.177011887s) [noise=true , address=77.90.31.14:4664 , piece_id=DEUYUFUOQ5ZJ64MC47P33XFP5WSDFQ35LUAG7JUOLRR64XAQLGMQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.310305796 (501h24m33.181388851s) [noise=true , address=145.79.227.52:4446 , piece_id=GE63KAOXOW5OQOJFQKD2OCRW2NHRWQ7ULBEVNW5OCVCVQFQ5N72Q, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.310942769 (501h24m33.182025814s) [noise=true , address=45.91.116.6:4976 , piece_id=Z64GC2HQ6FXCJKX7ITZTWRIUNMAYGOIYHAXGTK42P6PJMG235DPQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.311429736 (501h24m33.182512781s) [noise=true , address=145.79.227.35:4481 , piece_id=KPKOFEE7P64X6K64CATZCX2PWSGQA4VVMC2TFQTBSSO7JZNF3QKA, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.311996105 (501h24m33.18307915s) [noise=true , address=145.79.227.45:4230 , piece_id=7NZXREFL4LBCBY6JZVGJSJFS2NYKVPSIQ4PE727BDANLV5ZSPGYQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.312483022 (501h24m33.183566067s) [noise=true , address=145.79.227.49:4337 , piece_id=YXIJMAZVJIEVFIXLRKMG3Z3BT2UCSGM2KUAUNTYWVMTDGKPD6K2Q, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.313996513 (501h24m33.185079558s) [noise=true , address=145.79.227.41:4086 , piece_id=3KQWI5WAPGUNIHIR24NKYKXQ2IDTBX4G6ZZX3UKHZSYHZKFSUAOQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.315528761 (501h24m33.186611806s) [noise=true , address=145.79.227.39:4013 , piece_id=BBAUK5WO4SYS7ZQYZWVYIVLYONF2VLIYAX3Q7PXZBXAU2K5FGULQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.316907637 (501h24m33.187990682s) [noise=true , address=145.79.227.37:4567 , piece_id=LTGY5WRV5WFRAHFW2CT5EQZPRDXLUVCW7XMKMCIRHQIZXOG3BZ7Q, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.319905165 (501h24m33.19098822s) [noise=true , address=45.91.116.11:4117 , piece_id=XGOCYUWEFN5FKC2XLO22VM2EYOMQL5ELL6ZJMUFPBRETUYMF443Q, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.321664966 (501h24m33.192748001s) [noise=true , address=145.79.227.38:4979 , piece_id=3KSMBSFYPFHKW55E4LWN6AUH64NPSYKI5ZXOTKM4ZG4XOYQHAVFQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:34.914820462 (501h24m34.785903507s) [upload=440045]: Next returned nil segment Dec 26 12:53:34.914837635 (501h24m34.78592069s) [upload=440045]: Waiting for error group managing segments... Dec 26 12:53:35.298879578 (501h24m35.169962623s) [noise=true , address=145.79.227.46:4602 , piece_id=734WH2AFE2ZXM7OEWH55N2YCJGKYA7QJC2UAEKG7S27K7O4AAICQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.308194576 (501h24m35.179277611s) [noise=true , address=145.79.227.4:4185 , piece_id=5FEZC7ARULDUTJGMZHLHO7NBQBV6NJJWIRHR74XKUJDD544W47OA, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.321282087 (501h24m35.192365122s) [noise=true , address=145.79.227.49:4337 , piece_id=YXIJMAZVJIEVFIXLRKMG3Z3BT2UCSGM2KUAUNTYWVMTDGKPD6K2Q, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.324654759 (501h24m35.195737804s) [noise=true , address=145.79.227.37:4567 , piece_id=LTGY5WRV5WFRAHFW2CT5EQZPRDXLUVCW7XMKMCIRHQIZXOG3BZ7Q, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.328057719 (501h24m35.199140754s) [noise=true , address=145.79.227.45:4230 , piece_id=7NZXREFL4LBCBY6JZVGJSJFS2NYKVPSIQ4PE727BDANLV5ZSPGYQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.336066038 (501h24m35.207149073s) [noise=true , address=145.79.227.38:4979 , piece_id=3KSMBSFYPFHKW55E4LWN6AUH64NPSYKI5ZXOTKM4ZG4XOYQHAVFQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.336911889 (501h24m35.207994934s) [noise=true , address=145.79.227.16:4910 , piece_id=VMMXT72NDQ4QZG6ELIFOCX4DGTYDLU4YMRWJTZAA46VDTMEAG6YA, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.344439162 (501h24m35.215522227s) [noise=true , address=145.79.227.41:4086 , piece_id=3KQWI5WAPGUNIHIR24NKYKXQ2IDTBX4G6ZZX3UKHZSYHZKFSUAOQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.344909568 (501h24m35.215992603s) [noise=true , address=145.79.227.39:4013 , piece_id=BBAUK5WO4SYS7ZQYZWVYIVLYONF2VLIYAX3Q7PXZBXAU2K5FGULQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.351561845 (501h24m35.22264488s) [noise=true , address=145.79.227.35:4481 , piece_id=KPKOFEE7P64X6K64CATZCX2PWSGQA4VVMC2TFQTBSSO7JZNF3QKA, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.352409338 (501h24m35.223492383s) [noise=true , address=145.79.227.52:4446 , piece_id=GE63KAOXOW5OQOJFQKD2OCRW2NHRWQ7ULBEVNW5OCVCVQFQ5N72Q, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.601492897 (501h24m35.472575942s) [noise=true , address=45.91.116.14:4189 , piece_id=FNMHETMVEEU7YVY5Z527MK6CTPI5TSXYTDIRP5OMFS3YSG47EEHQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.610741338 (501h24m35.481824383s) [noise=true , address=45.91.116.13:4154 , piece_id=4AQ7JCS2YYX3BUINUIE375WQFUU6VNBADRTYAHITMK724H5QUARQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.610765683 (501h24m35.481848729s) [seg_pos={0 0}, upload=440045]: Segment reached optimal threshold of 13 pieces. Dec 26 12:53:35.610962629 (501h24m35.482045674s) [noise=true , address=45.91.116.11:4117 , piece_id=XGOCYUWEFN5FKC2XLO22VM2EYOMQL5ELL6ZJMUFPBRETUYMF443Q, seg_pos={0 0}, upload=440045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 26 12:53:35.610968921 (501h24m35.482051956s) [noise=true , address=45.91.116.11:4117 , piece_id=XGOCYUWEFN5FKC2XLO22VM2EYOMQL5ELL6ZJMUFPBRETUYMF443Q, seg_pos={0 0}, upload=440045]: Long tail cancellation due to optimal threshold Dec 26 12:53:35.611078359 (501h24m35.482161394s) [noise=true , address=77.90.31.14:4664 , piece_id=DEUYUFUOQ5ZJ64MC47P33XFP5WSDFQ35LUAG7JUOLRR64XAQLGMQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: write tcp 188.166.225.170:54322->77.90.31.14:4664: use of closed network connection; write tcp 188.166.225.170:54322->77.90.31.14:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54322->77.90.31.14:4664: use of closed network connection Dec 26 12:53:35.611088449 (501h24m35.482171494s) [noise=true , address=77.90.31.14:4664 , piece_id=DEUYUFUOQ5ZJ64MC47P33XFP5WSDFQ35LUAG7JUOLRR64XAQLGMQ, seg_pos={0 0}, upload=440045]: Long tail cancellation due to optimal threshold Dec 26 12:53:35.611146599 (501h24m35.482229634s) [noise=true , address=45.91.116.6:4976 , piece_id=Z64GC2HQ6FXCJKX7ITZTWRIUNMAYGOIYHAXGTK42P6PJMG235DPQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:53846->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:53846->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53846->45.91.116.6:4976: use of closed network connection Dec 26 12:53:35.611152770 (501h24m35.482235816s) [noise=true , address=45.91.116.6:4976 , piece_id=Z64GC2HQ6FXCJKX7ITZTWRIUNMAYGOIYHAXGTK42P6PJMG235DPQ, seg_pos={0 0}, upload=440045]: Long tail cancellation due to optimal threshold Dec 26 12:53:35.611193359 (501h24m35.482276404s) [seg_pos={0 0}, upload=440045]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:53:35.611282528 (501h24m35.482365583s) [upload=440045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:53:35.611289651 (501h24m35.482372686s) [upload=440045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:53:35.896282848 (501h24m35.767365894s) [upload=440045]: Done uploading segments... Dec 26 12:53:35.896302896 (501h24m35.767385931s) [upload=440045]: Upload finished. err: <nil> Duration: 5.898272643s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [........................X....................X.XXX...X..X...] duration: 39.659163378s trace: https://jaeger.dev.storj.io/trace/4fdd8919ff1f03ab FAIL (2025-12-26 12:54:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 64.143:4695 , piece_id=MJU7BWFRV3XEUAUXWLV3PILJFJ3EEYU5JQDR7ELFP56K7UMOJ5SQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.504565702 (501h21m41.375648747s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.504919806 (501h21m41.376002841s) [noise=true , address=77.90.31.13:4645 , piece_id=H77LB7OQOYZWJ5TM6FOXIYZEYPH5XKTAVEIE7DXNMHQM3EAB4G3Q, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.505319897 (501h21m41.376402942s) [noise=true , address=77.90.31.18:4822 , piece_id=BEBBCJKIZXWEPGKWT4CKULV4SUZQIS5YZVO4U7FSMICKI7KA42QQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.505772810 (501h21m41.376855855s) [noise=true , address=77.90.31.11:4566 , piece_id=VBI6QVPQTFN4JX7FBFNYHL4OYHBX4MPTIQDQGHBFICF5SBZ45XPQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.506212748 (501h21m41.377295793s) [noise=true , address=145.79.227.40:4081 , piece_id=TLYGZBCLBK5QYAX2O3CAEPQQW6GHSNOCWVMCJK64PUF2ZVB3QZOQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.506727949 (501h21m41.377810994s) [noise=true , address=77.90.31.15:4702 , piece_id=J3U7MZX4Y4QVWWDLEZL73RTSV3PY3KJJM5V22LIJ5KW3GNF3A4WQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.508679004 (501h21m41.379762039s) [noise=true , address=77.90.31.17:4777 , piece_id=M5T57WEHR6GEA2K4LUKPHTVNJXOB4N44OQQYSPUZBYBSZNN4LUGA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.510244746 (501h21m41.381327791s) [noise=true , address=45.91.116.8:4062 , piece_id=ZIQQ76IAON3KH3NPW3OLJNDDZ67U3PDUYFKKDF6WB74EBRONATPQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.513404332 (501h21m41.384487367s) [noise=true , address=64.49.13.138:4481 , piece_id=AYNG7FPLCAWWPCQOREPGIKVMHYRW7UHQC6W2M6GD3SYSKNQZB2FA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.515190553 (501h21m41.386273588s) [noise=true , address=145.79.227.17:4002 , piece_id=L73THH6HKT6X2U3UVHRF56DZJXRN56XEQBYCEQRMK6XPZTH5G5BQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.516689447 (501h21m41.387772482s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.518014290 (501h21m41.389097335s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.519302834 (501h21m41.390385879s) [noise=true , address=145.79.227.34:4918 , piece_id=NQSH2PFCL4NXVGTYWT2CSBIV3TQSEEBDUIPQBI33ZJVHVP7DWHIQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.520341321 (501h21m41.391424366s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:42.797832918 (501h21m42.668915953s) [upload=440005]: Next returned nil segment Dec 26 12:50:42.797878615 (501h21m42.66896167s) [upload=440005]: Waiting for error group managing segments... Dec 26 12:50:43.150930207 (501h21m43.022013273s) [noise=true , address=64.49.13.138:4481 , piece_id=AYNG7FPLCAWWPCQOREPGIKVMHYRW7UHQC6W2M6GD3SYSKNQZB2FA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.162932632 (501h21m43.034015667s) [noise=true , address=193.149.164.143:4695 , piece_id=MJU7BWFRV3XEUAUXWLV3PILJFJ3EEYU5JQDR7ELFP56K7UMOJ5SQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.374778985 (501h21m43.24586202s) [noise=true , address=145.79.227.40:4081 , piece_id=TLYGZBCLBK5QYAX2O3CAEPQQW6GHSNOCWVMCJK64PUF2ZVB3QZOQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.375783878 (501h21m43.246866913s) [noise=true , address=145.79.227.23:4580 , piece_id=YANWHT6JCLD6JNE7EFJFRZXHNKNKRTUXHEU2NHQLK5U4FNB7JR7A, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.391967825 (501h21m43.263050871s) [noise=true , address=145.79.227.34:4918 , piece_id=NQSH2PFCL4NXVGTYWT2CSBIV3TQSEEBDUIPQBI33ZJVHVP7DWHIQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.392038590 (501h21m43.263121625s) [noise=true , address=145.79.227.52:4476 , piece_id=B7F5LMSUB26MRAIRWA2U5J5IFMZGXDDJUZZKPA6PFJ6WQQ5QGCCQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.405638596 (501h21m43.276721641s) [noise=true , address=145.79.227.17:4002 , piece_id=L73THH6HKT6X2U3UVHRF56DZJXRN56XEQBYCEQRMK6XPZTH5G5BQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.599725422 (501h21m43.470808477s) [noise=true , address=45.91.116.6:4002 , piece_id=UYE5IA3ZW7ICL2F7ALYTXFDMR53CAXNHBKIK34JK6IBQCJJRF3AA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.638993344 (501h21m43.510076389s) [noise=true , address=45.91.116.8:4062 , piece_id=ZIQQ76IAON3KH3NPW3OLJNDDZ67U3PDUYFKKDF6WB74EBRONATPQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.662919455 (501h21m43.53400249s) [noise=true , address=77.90.31.18:4822 , piece_id=BEBBCJKIZXWEPGKWT4CKULV4SUZQIS5YZVO4U7FSMICKI7KA42QQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.687034162 (501h21m43.558117207s) [noise=true , address=77.90.31.11:4566 , piece_id=VBI6QVPQTFN4JX7FBFNYHL4OYHBX4MPTIQDQGHBFICF5SBZ45XPQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.767762384 (501h21m43.638845429s) [noise=true , address=77.90.31.15:4702 , piece_id=J3U7MZX4Y4QVWWDLEZL73RTSV3PY3KJJM5V22LIJ5KW3GNF3A4WQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.769485905 (501h21m43.64056894s) [noise=true , address=77.90.31.17:4777 , piece_id=M5T57WEHR6GEA2K4LUKPHTVNJXOB4N44OQQYSPUZBYBSZNN4LUGA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.882129930 (501h21m43.753212975s) [noise=true , address=77.90.31.13:4645 , piece_id=H77LB7OQOYZWJ5TM6FOXIYZEYPH5XKTAVEIE7DXNMHQM3EAB4G3Q, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.882154297 (501h21m43.753237332s) [seg_pos={0 0}, upload=440005]: Segment reached optimal threshold of 14 pieces. Dec 26 12:50:43.882433248 (501h21m43.753516283s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 26 12:50:43.882442595 (501h21m43.75352565s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882668485 (501h21m43.75375153s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection Dec 26 12:50:43.882679026 (501h21m43.753762061s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882691860 (501h21m43.753774905s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj): protocol: expected piece hash; context canceled; EOF Dec 26 12:50:43.882699975 (501h21m43.75378302s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882859289 (501h21m43.753942334s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121bMz2UYi9i53u7ss3SULotcCmAQVhXbNPFr4Hk7XGdgHeVQMw): protocol: write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection; write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection Dec 26 12:50:43.882867174 (501h21m43.753950219s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882886611 (501h21m43.753969646s) [seg_pos={0 0}, upload=440005]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 12:50:43.882980860 (501h21m43.754063895s) [upload=440005]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:50:43.882987884 (501h21m43.754070929s) [upload=440005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:50:44.189484256 (501h21m44.060567291s) [upload=440005]: Done uploading segments... Dec 26 12:50:44.189501439 (501h21m44.060584474s) [upload=440005]: Upload finished. err: <nil> Duration: 39.659191693s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X...X..XX.....XXXX.....XXX....XX....X..X......X.X..XX.....X] duration: 30.004943123s trace: https://jaeger.dev.storj.io/trace/25f0a86b327adac5 FAIL (2025-12-26 12:54:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: 4AQ7JCS2YYX3BUINUIE375WQFUU6VNBADRTYAHITMK724H5QUARQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: FNMHETMVEEU7YVY5Z527MK6CTPI5TSXYTDIRP5OMFS3YSG47EEHQ) context deadline exceeded ======LOGS====== segments... Dec 26 12:53:33.060139405 (501h24m32.93122247s) [upload=440045]: Got next segment. Inline: false Dec 26 12:53:33.060183419 (501h24m32.931266454s) [upload=440045]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:53:33.060189249 (501h24m32.931272295s) [upload=440045]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:53:33.304580344 (501h24m33.175663389s) [seg_pos={0 0}, upload=440045]: Begin upload segment... Dec 26 12:53:33.304632342 (501h24m33.175715377s) [seg_pos={0 0}, upload=440045]: Done begin upload segment. Dec 26 12:53:33.304685023 (501h24m33.175768068s) [noise=true , address=145.79.227.16:4910 , piece_id=VMMXT72NDQ4QZG6ELIFOCX4DGTYDLU4YMRWJTZAA46VDTMEAG6YA, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.304695913 (501h24m33.175778959s) [noise=true , address=45.91.116.13:4154 , piece_id=4AQ7JCS2YYX3BUINUIE375WQFUU6VNBADRTYAHITMK724H5QUARQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.304862942 (501h24m33.175945977s) [noise=true , address=45.91.116.14:4189 , piece_id=FNMHETMVEEU7YVY5Z527MK6CTPI5TSXYTDIRP5OMFS3YSG47EEHQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.305344799 (501h24m33.176427844s) [noise=true , address=145.79.227.4:4185 , piece_id=5FEZC7ARULDUTJGMZHLHO7NBQBV6NJJWIRHR74XKUJDD544W47OA, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.305502649 (501h24m33.176585685s) [noise=true , address=145.79.227.46:4602 , piece_id=734WH2AFE2ZXM7OEWH55N2YCJGKYA7QJC2UAEKG7S27K7O4AAICQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.305928852 (501h24m33.177011887s) [noise=true , address=77.90.31.14:4664 , piece_id=DEUYUFUOQ5ZJ64MC47P33XFP5WSDFQ35LUAG7JUOLRR64XAQLGMQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.310305796 (501h24m33.181388851s) [noise=true , address=145.79.227.52:4446 , piece_id=GE63KAOXOW5OQOJFQKD2OCRW2NHRWQ7ULBEVNW5OCVCVQFQ5N72Q, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.310942769 (501h24m33.182025814s) [noise=true , address=45.91.116.6:4976 , piece_id=Z64GC2HQ6FXCJKX7ITZTWRIUNMAYGOIYHAXGTK42P6PJMG235DPQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.311429736 (501h24m33.182512781s) [noise=true , address=145.79.227.35:4481 , piece_id=KPKOFEE7P64X6K64CATZCX2PWSGQA4VVMC2TFQTBSSO7JZNF3QKA, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.311996105 (501h24m33.18307915s) [noise=true , address=145.79.227.45:4230 , piece_id=7NZXREFL4LBCBY6JZVGJSJFS2NYKVPSIQ4PE727BDANLV5ZSPGYQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.312483022 (501h24m33.183566067s) [noise=true , address=145.79.227.49:4337 , piece_id=YXIJMAZVJIEVFIXLRKMG3Z3BT2UCSGM2KUAUNTYWVMTDGKPD6K2Q, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.313996513 (501h24m33.185079558s) [noise=true , address=145.79.227.41:4086 , piece_id=3KQWI5WAPGUNIHIR24NKYKXQ2IDTBX4G6ZZX3UKHZSYHZKFSUAOQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.315528761 (501h24m33.186611806s) [noise=true , address=145.79.227.39:4013 , piece_id=BBAUK5WO4SYS7ZQYZWVYIVLYONF2VLIYAX3Q7PXZBXAU2K5FGULQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.316907637 (501h24m33.187990682s) [noise=true , address=145.79.227.37:4567 , piece_id=LTGY5WRV5WFRAHFW2CT5EQZPRDXLUVCW7XMKMCIRHQIZXOG3BZ7Q, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.319905165 (501h24m33.19098822s) [noise=true , address=45.91.116.11:4117 , piece_id=XGOCYUWEFN5FKC2XLO22VM2EYOMQL5ELL6ZJMUFPBRETUYMF443Q, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:33.321664966 (501h24m33.192748001s) [noise=true , address=145.79.227.38:4979 , piece_id=3KSMBSFYPFHKW55E4LWN6AUH64NPSYKI5ZXOTKM4ZG4XOYQHAVFQ, seg_pos={0 0}, upload=440045]: Uploading piece... Dec 26 12:53:34.914820462 (501h24m34.785903507s) [upload=440045]: Next returned nil segment Dec 26 12:53:34.914837635 (501h24m34.78592069s) [upload=440045]: Waiting for error group managing segments... Dec 26 12:53:35.298879578 (501h24m35.169962623s) [noise=true , address=145.79.227.46:4602 , piece_id=734WH2AFE2ZXM7OEWH55N2YCJGKYA7QJC2UAEKG7S27K7O4AAICQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.308194576 (501h24m35.179277611s) [noise=true , address=145.79.227.4:4185 , piece_id=5FEZC7ARULDUTJGMZHLHO7NBQBV6NJJWIRHR74XKUJDD544W47OA, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.321282087 (501h24m35.192365122s) [noise=true , address=145.79.227.49:4337 , piece_id=YXIJMAZVJIEVFIXLRKMG3Z3BT2UCSGM2KUAUNTYWVMTDGKPD6K2Q, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.324654759 (501h24m35.195737804s) [noise=true , address=145.79.227.37:4567 , piece_id=LTGY5WRV5WFRAHFW2CT5EQZPRDXLUVCW7XMKMCIRHQIZXOG3BZ7Q, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.328057719 (501h24m35.199140754s) [noise=true , address=145.79.227.45:4230 , piece_id=7NZXREFL4LBCBY6JZVGJSJFS2NYKVPSIQ4PE727BDANLV5ZSPGYQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.336066038 (501h24m35.207149073s) [noise=true , address=145.79.227.38:4979 , piece_id=3KSMBSFYPFHKW55E4LWN6AUH64NPSYKI5ZXOTKM4ZG4XOYQHAVFQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.336911889 (501h24m35.207994934s) [noise=true , address=145.79.227.16:4910 , piece_id=VMMXT72NDQ4QZG6ELIFOCX4DGTYDLU4YMRWJTZAA46VDTMEAG6YA, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.344439162 (501h24m35.215522227s) [noise=true , address=145.79.227.41:4086 , piece_id=3KQWI5WAPGUNIHIR24NKYKXQ2IDTBX4G6ZZX3UKHZSYHZKFSUAOQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.344909568 (501h24m35.215992603s) [noise=true , address=145.79.227.39:4013 , piece_id=BBAUK5WO4SYS7ZQYZWVYIVLYONF2VLIYAX3Q7PXZBXAU2K5FGULQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.351561845 (501h24m35.22264488s) [noise=true , address=145.79.227.35:4481 , piece_id=KPKOFEE7P64X6K64CATZCX2PWSGQA4VVMC2TFQTBSSO7JZNF3QKA, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.352409338 (501h24m35.223492383s) [noise=true , address=145.79.227.52:4446 , piece_id=GE63KAOXOW5OQOJFQKD2OCRW2NHRWQ7ULBEVNW5OCVCVQFQ5N72Q, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.601492897 (501h24m35.472575942s) [noise=true , address=45.91.116.14:4189 , piece_id=FNMHETMVEEU7YVY5Z527MK6CTPI5TSXYTDIRP5OMFS3YSG47EEHQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.610741338 (501h24m35.481824383s) [noise=true , address=45.91.116.13:4154 , piece_id=4AQ7JCS2YYX3BUINUIE375WQFUU6VNBADRTYAHITMK724H5QUARQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: <nil> Dec 26 12:53:35.610765683 (501h24m35.481848729s) [seg_pos={0 0}, upload=440045]: Segment reached optimal threshold of 13 pieces. Dec 26 12:53:35.610962629 (501h24m35.482045674s) [noise=true , address=45.91.116.11:4117 , piece_id=XGOCYUWEFN5FKC2XLO22VM2EYOMQL5ELL6ZJMUFPBRETUYMF443Q, seg_pos={0 0}, upload=440045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 26 12:53:35.610968921 (501h24m35.482051956s) [noise=true , address=45.91.116.11:4117 , piece_id=XGOCYUWEFN5FKC2XLO22VM2EYOMQL5ELL6ZJMUFPBRETUYMF443Q, seg_pos={0 0}, upload=440045]: Long tail cancellation due to optimal threshold Dec 26 12:53:35.611078359 (501h24m35.482161394s) [noise=true , address=77.90.31.14:4664 , piece_id=DEUYUFUOQ5ZJ64MC47P33XFP5WSDFQ35LUAG7JUOLRR64XAQLGMQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: write tcp 188.166.225.170:54322->77.90.31.14:4664: use of closed network connection; write tcp 188.166.225.170:54322->77.90.31.14:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54322->77.90.31.14:4664: use of closed network connection Dec 26 12:53:35.611088449 (501h24m35.482171494s) [noise=true , address=77.90.31.14:4664 , piece_id=DEUYUFUOQ5ZJ64MC47P33XFP5WSDFQ35LUAG7JUOLRR64XAQLGMQ, seg_pos={0 0}, upload=440045]: Long tail cancellation due to optimal threshold Dec 26 12:53:35.611146599 (501h24m35.482229634s) [noise=true , address=45.91.116.6:4976 , piece_id=Z64GC2HQ6FXCJKX7ITZTWRIUNMAYGOIYHAXGTK42P6PJMG235DPQ, seg_pos={0 0}, upload=440045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: write tcp 188.166.225.170:53846->45.91.116.6:4976: use of closed network connection; write tcp 188.166.225.170:53846->45.91.116.6:4976: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53846->45.91.116.6:4976: use of closed network connection Dec 26 12:53:35.611152770 (501h24m35.482235816s) [noise=true , address=45.91.116.6:4976 , piece_id=Z64GC2HQ6FXCJKX7ITZTWRIUNMAYGOIYHAXGTK42P6PJMG235DPQ, seg_pos={0 0}, upload=440045]: Long tail cancellation due to optimal threshold Dec 26 12:53:35.611193359 (501h24m35.482276404s) [seg_pos={0 0}, upload=440045]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:53:35.611282528 (501h24m35.482365583s) [upload=440045]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:53:35.611289651 (501h24m35.482372686s) [upload=440045]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:53:35.896282848 (501h24m35.767365894s) [upload=440045]: Done uploading segments... Dec 26 12:53:35.896302896 (501h24m35.767385931s) [upload=440045]: Upload finished. err: <nil> Duration: 30.004969061s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [.........................X....................X.XXX...X..X..] duration: 44.881595442s trace: https://jaeger.dev.storj.io/trace/4fdd8919ff1f03ab FAIL (2025-12-26 12:53:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 64.143:4695 , piece_id=MJU7BWFRV3XEUAUXWLV3PILJFJ3EEYU5JQDR7ELFP56K7UMOJ5SQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.504565702 (501h21m41.375648747s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.504919806 (501h21m41.376002841s) [noise=true , address=77.90.31.13:4645 , piece_id=H77LB7OQOYZWJ5TM6FOXIYZEYPH5XKTAVEIE7DXNMHQM3EAB4G3Q, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.505319897 (501h21m41.376402942s) [noise=true , address=77.90.31.18:4822 , piece_id=BEBBCJKIZXWEPGKWT4CKULV4SUZQIS5YZVO4U7FSMICKI7KA42QQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.505772810 (501h21m41.376855855s) [noise=true , address=77.90.31.11:4566 , piece_id=VBI6QVPQTFN4JX7FBFNYHL4OYHBX4MPTIQDQGHBFICF5SBZ45XPQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.506212748 (501h21m41.377295793s) [noise=true , address=145.79.227.40:4081 , piece_id=TLYGZBCLBK5QYAX2O3CAEPQQW6GHSNOCWVMCJK64PUF2ZVB3QZOQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.506727949 (501h21m41.377810994s) [noise=true , address=77.90.31.15:4702 , piece_id=J3U7MZX4Y4QVWWDLEZL73RTSV3PY3KJJM5V22LIJ5KW3GNF3A4WQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.508679004 (501h21m41.379762039s) [noise=true , address=77.90.31.17:4777 , piece_id=M5T57WEHR6GEA2K4LUKPHTVNJXOB4N44OQQYSPUZBYBSZNN4LUGA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.510244746 (501h21m41.381327791s) [noise=true , address=45.91.116.8:4062 , piece_id=ZIQQ76IAON3KH3NPW3OLJNDDZ67U3PDUYFKKDF6WB74EBRONATPQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.513404332 (501h21m41.384487367s) [noise=true , address=64.49.13.138:4481 , piece_id=AYNG7FPLCAWWPCQOREPGIKVMHYRW7UHQC6W2M6GD3SYSKNQZB2FA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.515190553 (501h21m41.386273588s) [noise=true , address=145.79.227.17:4002 , piece_id=L73THH6HKT6X2U3UVHRF56DZJXRN56XEQBYCEQRMK6XPZTH5G5BQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.516689447 (501h21m41.387772482s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.518014290 (501h21m41.389097335s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.519302834 (501h21m41.390385879s) [noise=true , address=145.79.227.34:4918 , piece_id=NQSH2PFCL4NXVGTYWT2CSBIV3TQSEEBDUIPQBI33ZJVHVP7DWHIQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.520341321 (501h21m41.391424366s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:42.797832918 (501h21m42.668915953s) [upload=440005]: Next returned nil segment Dec 26 12:50:42.797878615 (501h21m42.66896167s) [upload=440005]: Waiting for error group managing segments... Dec 26 12:50:43.150930207 (501h21m43.022013273s) [noise=true , address=64.49.13.138:4481 , piece_id=AYNG7FPLCAWWPCQOREPGIKVMHYRW7UHQC6W2M6GD3SYSKNQZB2FA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.162932632 (501h21m43.034015667s) [noise=true , address=193.149.164.143:4695 , piece_id=MJU7BWFRV3XEUAUXWLV3PILJFJ3EEYU5JQDR7ELFP56K7UMOJ5SQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.374778985 (501h21m43.24586202s) [noise=true , address=145.79.227.40:4081 , piece_id=TLYGZBCLBK5QYAX2O3CAEPQQW6GHSNOCWVMCJK64PUF2ZVB3QZOQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.375783878 (501h21m43.246866913s) [noise=true , address=145.79.227.23:4580 , piece_id=YANWHT6JCLD6JNE7EFJFRZXHNKNKRTUXHEU2NHQLK5U4FNB7JR7A, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.391967825 (501h21m43.263050871s) [noise=true , address=145.79.227.34:4918 , piece_id=NQSH2PFCL4NXVGTYWT2CSBIV3TQSEEBDUIPQBI33ZJVHVP7DWHIQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.392038590 (501h21m43.263121625s) [noise=true , address=145.79.227.52:4476 , piece_id=B7F5LMSUB26MRAIRWA2U5J5IFMZGXDDJUZZKPA6PFJ6WQQ5QGCCQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.405638596 (501h21m43.276721641s) [noise=true , address=145.79.227.17:4002 , piece_id=L73THH6HKT6X2U3UVHRF56DZJXRN56XEQBYCEQRMK6XPZTH5G5BQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.599725422 (501h21m43.470808477s) [noise=true , address=45.91.116.6:4002 , piece_id=UYE5IA3ZW7ICL2F7ALYTXFDMR53CAXNHBKIK34JK6IBQCJJRF3AA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.638993344 (501h21m43.510076389s) [noise=true , address=45.91.116.8:4062 , piece_id=ZIQQ76IAON3KH3NPW3OLJNDDZ67U3PDUYFKKDF6WB74EBRONATPQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.662919455 (501h21m43.53400249s) [noise=true , address=77.90.31.18:4822 , piece_id=BEBBCJKIZXWEPGKWT4CKULV4SUZQIS5YZVO4U7FSMICKI7KA42QQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.687034162 (501h21m43.558117207s) [noise=true , address=77.90.31.11:4566 , piece_id=VBI6QVPQTFN4JX7FBFNYHL4OYHBX4MPTIQDQGHBFICF5SBZ45XPQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.767762384 (501h21m43.638845429s) [noise=true , address=77.90.31.15:4702 , piece_id=J3U7MZX4Y4QVWWDLEZL73RTSV3PY3KJJM5V22LIJ5KW3GNF3A4WQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.769485905 (501h21m43.64056894s) [noise=true , address=77.90.31.17:4777 , piece_id=M5T57WEHR6GEA2K4LUKPHTVNJXOB4N44OQQYSPUZBYBSZNN4LUGA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.882129930 (501h21m43.753212975s) [noise=true , address=77.90.31.13:4645 , piece_id=H77LB7OQOYZWJ5TM6FOXIYZEYPH5XKTAVEIE7DXNMHQM3EAB4G3Q, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.882154297 (501h21m43.753237332s) [seg_pos={0 0}, upload=440005]: Segment reached optimal threshold of 14 pieces. Dec 26 12:50:43.882433248 (501h21m43.753516283s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 26 12:50:43.882442595 (501h21m43.75352565s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882668485 (501h21m43.75375153s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection Dec 26 12:50:43.882679026 (501h21m43.753762061s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882691860 (501h21m43.753774905s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj): protocol: expected piece hash; context canceled; EOF Dec 26 12:50:43.882699975 (501h21m43.75378302s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882859289 (501h21m43.753942334s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121bMz2UYi9i53u7ss3SULotcCmAQVhXbNPFr4Hk7XGdgHeVQMw): protocol: write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection; write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection Dec 26 12:50:43.882867174 (501h21m43.753950219s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882886611 (501h21m43.753969646s) [seg_pos={0 0}, upload=440005]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 12:50:43.882980860 (501h21m43.754063895s) [upload=440005]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:50:43.882987884 (501h21m43.754070929s) [upload=440005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:50:44.189484256 (501h21m44.060567291s) [upload=440005]: Done uploading segments... Dec 26 12:50:44.189501439 (501h21m44.060584474s) [upload=440005]: Upload finished. err: <nil> Duration: 44.881624407s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [....X.......XXX...X.....X.......X.XX......XXX..XXXXXXX.XXXX.] duration: 3.531158312s trace: https://jaeger.dev.storj.io/trace/f45800a7d94cace FAIL (2025-12-26 12:53:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.531210682s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..X...X..XX.....XXXX.....XXX....XX....X..X......X.X..XX.....] duration: 7.740231645s trace: https://jaeger.dev.storj.io/trace/a7946c4655f3019 FAIL (2025-12-26 12:52:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: 67B4CJKGQOCJNRJK3MKI3T6B4I6A6BFRIS4ZIU5LOATZWTZ5O3TA) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: KAGOUHT7WVAZFM3Y4RIAWDVDHF4WB3TS2IGMX2OU74B4P4B3ZXRA) context deadline exceeded ======LOGS====== Dec 26 12:49:02.447043594 (501h20m2.318126639s) [upload=439980]: Starting upload Dec 26 12:49:02.447156158 (501h20m2.318239203s) [upload=439980]: Done starting upload Dec 26 12:49:02.447204801 (501h20m2.318287846s) [upload=439980]: Uploading segments... Dec 26 12:49:02.447398079 (501h20m2.318481124s) [upload=439980]: Got next segment. Inline: false Dec 26 12:49:02.447447503 (501h20m2.318530538s) [upload=439980]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:49:02.447454657 (501h20m2.318537702s) [upload=439980]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:49:02.688830688 (501h20m2.559913744s) [seg_pos={0 0}, upload=439980]: Begin upload segment... Dec 26 12:49:02.688890753 (501h20m2.559973788s) [seg_pos={0 0}, upload=439980]: Done begin upload segment. Dec 26 12:49:02.688920770 (501h20m2.560003815s) [noise=true , address=145.79.227.45:4229 , piece_id=TRG4WAIVFL33BVBJ5PCIQ5EUPZ4LI5UL2TJLK6EN5HMVLD5J5EMQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.689093059 (501h20m2.560176094s) [noise=true , address=145.79.227.46:4602 , piece_id=CPIT2K2YAAC55GDEC5LPCNJ22SSII7MVTBUORL6X2HACU2CKOABQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.689721966 (501h20m2.560805021s) [noise=true , address=145.79.227.31:4799 , piece_id=SE75ODCMXKIUAR3JTTYRYSHF6BOWH6WMCXGRX4KWRHZPLFJDI57A, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.690107059 (501h20m2.561190104s) [noise=true , address=145.79.227.15:4876 , piece_id=2UCA4YCUUVYCOWZ4HZF3R2T7GT64Z5KZKL6N6QXVMZR26MVGDR3A, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.690704497 (501h20m2.561787542s) [noise=true , address=145.79.227.41:4087 , piece_id=L77EWW6ZI4P5KXRBD7D4WQKMZ3J5YO42LUI53FBIAOQRT4D2X5XQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.691216792 (501h20m2.562299838s) [noise=true , address=145.79.227.17:4971 , piece_id=ZEELPS5ZQTMLZ2LIPNUJ4XRV7TID4VOYPJXVUOL3DA5R3LMPU2VQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.691700504 (501h20m2.562783549s) [noise=true , address=45.91.116.14:4191 , piece_id=ANVMW7PZB54QZQCCIJTFZAMCLNHQP6M5B7SO2XDG72RNE2P5DA2Q, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.692170979 (501h20m2.563254024s) [noise=true , address=145.79.227.35:4481 , piece_id=HM2ZHXJJTEELPP76YNF3FV3YKQ4X64QJPOXDIPANOICYBJZNEW5Q, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.692728280 (501h20m2.563811325s) [noise=true , address=145.79.227.26:4654 , piece_id=QLPH4VGYQJGALFYUZOJ23OWOMOYNPIQ3MSYM6XQIA5ZFYTQ2RAWQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.694872544 (501h20m2.565955589s) [noise=true , address=145.79.227.11:4731 , piece_id=N6N2IEWXNY3B4ASST2B2EZ5375T2F6AV6JXELAYSRW4X2MCCGRTQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.697026275 (501h20m2.56810931s) [noise=true , address=45.91.116.11:4117 , piece_id=IENGJEVLVYZNAOBXBKFLAEJ3EOBYYSL7C3LGDYXYD52TGNR5R6EQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.699009491 (501h20m2.570092536s) [noise=true , address=77.90.31.16:4735 , piece_id=KAGOUHT7WVAZFM3Y4RIAWDVDHF4WB3TS2IGMX2OU74B4P4B3ZXRA, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.700818746 (501h20m2.571901791s) [noise=true , address=145.79.227.39:4014 , piece_id=VGHWBYL4OHJBVBCBPHL7DNY5O5C57OHL766P4JZEWRKPSBOEOONQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.703072998 (501h20m2.574156033s) [noise=true , address=77.90.31.10:4519 , piece_id=QCIWEIL43J22JV6ESHU2TPHBGBDPS2CMWUSVRKW4HL5XCETPQPCA, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.707037057 (501h20m2.578120102s) [noise=true , address=145.79.227.36:4518 , piece_id=URZLNXCY4VBUY4HSBWP6WKCM2RUMEGIL4UT7ZZM4QA6BJLEF7LAQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.708645239 (501h20m2.579728294s) [noise=true , address=77.90.31.14:4663 , piece_id=67B4CJKGQOCJNRJK3MKI3T6B4I6A6BFRIS4ZIU5LOATZWTZ5O3TA, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:04.416018446 (501h20m4.287101491s) [upload=439980]: Next returned nil segment Dec 26 12:49:04.416077238 (501h20m4.287160283s) [upload=439980]: Waiting for error group managing segments... Dec 26 12:49:04.703930962 (501h20m4.575014007s) [noise=true , address=145.79.227.15:4876 , piece_id=2UCA4YCUUVYCOWZ4HZF3R2T7GT64Z5KZKL6N6QXVMZR26MVGDR3A, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.704247905 (501h20m4.57533096s) [noise=true , address=145.79.227.35:4481 , piece_id=HM2ZHXJJTEELPP76YNF3FV3YKQ4X64QJPOXDIPANOICYBJZNEW5Q, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.707534473 (501h20m4.578617508s) [noise=true , address=145.79.227.26:4654 , piece_id=QLPH4VGYQJGALFYUZOJ23OWOMOYNPIQ3MSYM6XQIA5ZFYTQ2RAWQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.707569971 (501h20m4.578653006s) [noise=true , address=145.79.227.11:4731 , piece_id=N6N2IEWXNY3B4ASST2B2EZ5375T2F6AV6JXELAYSRW4X2MCCGRTQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.713069783 (501h20m4.584152828s) [noise=true , address=145.79.227.17:4971 , piece_id=ZEELPS5ZQTMLZ2LIPNUJ4XRV7TID4VOYPJXVUOL3DA5R3LMPU2VQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.715337060 (501h20m4.586420106s) [noise=true , address=145.79.227.46:4602 , piece_id=CPIT2K2YAAC55GDEC5LPCNJ22SSII7MVTBUORL6X2HACU2CKOABQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.719134893 (501h20m4.590217938s) [noise=true , address=145.79.227.45:4229 , piece_id=TRG4WAIVFL33BVBJ5PCIQ5EUPZ4LI5UL2TJLK6EN5HMVLD5J5EMQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.721251483 (501h20m4.592334518s) [noise=true , address=145.79.227.36:4518 , piece_id=URZLNXCY4VBUY4HSBWP6WKCM2RUMEGIL4UT7ZZM4QA6BJLEF7LAQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.721582844 (501h20m4.592665889s) [noise=true , address=145.79.227.41:4087 , piece_id=L77EWW6ZI4P5KXRBD7D4WQKMZ3J5YO42LUI53FBIAOQRT4D2X5XQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.728346563 (501h20m4.599429608s) [noise=true , address=145.79.227.31:4799 , piece_id=SE75ODCMXKIUAR3JTTYRYSHF6BOWH6WMCXGRX4KWRHZPLFJDI57A, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.733132587 (501h20m4.604215632s) [noise=true , address=145.79.227.39:4014 , piece_id=VGHWBYL4OHJBVBCBPHL7DNY5O5C57OHL766P4JZEWRKPSBOEOONQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.978493166 (501h20m4.849576202s) [noise=true , address=77.90.31.16:4735 , piece_id=KAGOUHT7WVAZFM3Y4RIAWDVDHF4WB3TS2IGMX2OU74B4P4B3ZXRA, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.978960205 (501h20m4.85004326s) [noise=true , address=77.90.31.14:4663 , piece_id=67B4CJKGQOCJNRJK3MKI3T6B4I6A6BFRIS4ZIU5LOATZWTZ5O3TA, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.978982949 (501h20m4.850065994s) [seg_pos={0 0}, upload=439980]: Segment reached optimal threshold of 13 pieces. Dec 26 12:49:04.979234939 (501h20m4.850317984s) [noise=true , address=45.91.116.11:4117 , piece_id=IENGJEVLVYZNAOBXBKFLAEJ3EOBYYSL7C3LGDYXYD52TGNR5R6EQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 26 12:49:04.979244006 (501h20m4.850327051s) [noise=true , address=45.91.116.11:4117 , piece_id=IENGJEVLVYZNAOBXBKFLAEJ3EOBYYSL7C3LGDYXYD52TGNR5R6EQ, seg_pos={0 0}, upload=439980]: Long tail cancellation due to optimal threshold Dec 26 12:49:04.979607709 (501h20m4.850690754s) [noise=true , address=77.90.31.10:4519 , piece_id=QCIWEIL43J22JV6ESHU2TPHBGBDPS2CMWUSVRKW4HL5XCETPQPCA, seg_pos={0 0}, upload=439980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 26 12:49:04.979615263 (501h20m4.850698298s) [noise=true , address=77.90.31.10:4519 , piece_id=QCIWEIL43J22JV6ESHU2TPHBGBDPS2CMWUSVRKW4HL5XCETPQPCA, seg_pos={0 0}, upload=439980]: Long tail cancellation due to optimal threshold Dec 26 12:49:04.979736964 (501h20m4.850820009s) [noise=true , address=45.91.116.14:4191 , piece_id=ANVMW7PZB54QZQCCIJTFZAMCLNHQP6M5B7SO2XDG72RNE2P5DA2Q, seg_pos={0 0}, upload=439980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:59208->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:59208->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59208->45.91.116.14:4191: use of closed network connection Dec 26 12:49:04.979748377 (501h20m4.850831412s) [noise=true , address=45.91.116.14:4191 , piece_id=ANVMW7PZB54QZQCCIJTFZAMCLNHQP6M5B7SO2XDG72RNE2P5DA2Q, seg_pos={0 0}, upload=439980]: Long tail cancellation due to optimal threshold Dec 26 12:49:04.979769035 (501h20m4.85085207s) [seg_pos={0 0}, upload=439980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:49:04.979846002 (501h20m4.850929047s) [upload=439980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:49:04.979851082 (501h20m4.850934117s) [upload=439980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:49:05.271932757 (501h20m5.143015792s) [upload=439980]: Done uploading segments... Dec 26 12:49:05.271960820 (501h20m5.143043855s) [upload=439980]: Upload finished. err: <nil> Duration: 7.740263296s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [..........................X....................X.XXX...X..X.] duration: 55.788700899s trace: https://jaeger.dev.storj.io/trace/4fdd8919ff1f03ab FAIL (2025-12-26 12:52:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 64.143:4695 , piece_id=MJU7BWFRV3XEUAUXWLV3PILJFJ3EEYU5JQDR7ELFP56K7UMOJ5SQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.504565702 (501h21m41.375648747s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.504919806 (501h21m41.376002841s) [noise=true , address=77.90.31.13:4645 , piece_id=H77LB7OQOYZWJ5TM6FOXIYZEYPH5XKTAVEIE7DXNMHQM3EAB4G3Q, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.505319897 (501h21m41.376402942s) [noise=true , address=77.90.31.18:4822 , piece_id=BEBBCJKIZXWEPGKWT4CKULV4SUZQIS5YZVO4U7FSMICKI7KA42QQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.505772810 (501h21m41.376855855s) [noise=true , address=77.90.31.11:4566 , piece_id=VBI6QVPQTFN4JX7FBFNYHL4OYHBX4MPTIQDQGHBFICF5SBZ45XPQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.506212748 (501h21m41.377295793s) [noise=true , address=145.79.227.40:4081 , piece_id=TLYGZBCLBK5QYAX2O3CAEPQQW6GHSNOCWVMCJK64PUF2ZVB3QZOQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.506727949 (501h21m41.377810994s) [noise=true , address=77.90.31.15:4702 , piece_id=J3U7MZX4Y4QVWWDLEZL73RTSV3PY3KJJM5V22LIJ5KW3GNF3A4WQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.508679004 (501h21m41.379762039s) [noise=true , address=77.90.31.17:4777 , piece_id=M5T57WEHR6GEA2K4LUKPHTVNJXOB4N44OQQYSPUZBYBSZNN4LUGA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.510244746 (501h21m41.381327791s) [noise=true , address=45.91.116.8:4062 , piece_id=ZIQQ76IAON3KH3NPW3OLJNDDZ67U3PDUYFKKDF6WB74EBRONATPQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.513404332 (501h21m41.384487367s) [noise=true , address=64.49.13.138:4481 , piece_id=AYNG7FPLCAWWPCQOREPGIKVMHYRW7UHQC6W2M6GD3SYSKNQZB2FA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.515190553 (501h21m41.386273588s) [noise=true , address=145.79.227.17:4002 , piece_id=L73THH6HKT6X2U3UVHRF56DZJXRN56XEQBYCEQRMK6XPZTH5G5BQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.516689447 (501h21m41.387772482s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.518014290 (501h21m41.389097335s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.519302834 (501h21m41.390385879s) [noise=true , address=145.79.227.34:4918 , piece_id=NQSH2PFCL4NXVGTYWT2CSBIV3TQSEEBDUIPQBI33ZJVHVP7DWHIQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.520341321 (501h21m41.391424366s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:42.797832918 (501h21m42.668915953s) [upload=440005]: Next returned nil segment Dec 26 12:50:42.797878615 (501h21m42.66896167s) [upload=440005]: Waiting for error group managing segments... Dec 26 12:50:43.150930207 (501h21m43.022013273s) [noise=true , address=64.49.13.138:4481 , piece_id=AYNG7FPLCAWWPCQOREPGIKVMHYRW7UHQC6W2M6GD3SYSKNQZB2FA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.162932632 (501h21m43.034015667s) [noise=true , address=193.149.164.143:4695 , piece_id=MJU7BWFRV3XEUAUXWLV3PILJFJ3EEYU5JQDR7ELFP56K7UMOJ5SQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.374778985 (501h21m43.24586202s) [noise=true , address=145.79.227.40:4081 , piece_id=TLYGZBCLBK5QYAX2O3CAEPQQW6GHSNOCWVMCJK64PUF2ZVB3QZOQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.375783878 (501h21m43.246866913s) [noise=true , address=145.79.227.23:4580 , piece_id=YANWHT6JCLD6JNE7EFJFRZXHNKNKRTUXHEU2NHQLK5U4FNB7JR7A, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.391967825 (501h21m43.263050871s) [noise=true , address=145.79.227.34:4918 , piece_id=NQSH2PFCL4NXVGTYWT2CSBIV3TQSEEBDUIPQBI33ZJVHVP7DWHIQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.392038590 (501h21m43.263121625s) [noise=true , address=145.79.227.52:4476 , piece_id=B7F5LMSUB26MRAIRWA2U5J5IFMZGXDDJUZZKPA6PFJ6WQQ5QGCCQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.405638596 (501h21m43.276721641s) [noise=true , address=145.79.227.17:4002 , piece_id=L73THH6HKT6X2U3UVHRF56DZJXRN56XEQBYCEQRMK6XPZTH5G5BQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.599725422 (501h21m43.470808477s) [noise=true , address=45.91.116.6:4002 , piece_id=UYE5IA3ZW7ICL2F7ALYTXFDMR53CAXNHBKIK34JK6IBQCJJRF3AA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.638993344 (501h21m43.510076389s) [noise=true , address=45.91.116.8:4062 , piece_id=ZIQQ76IAON3KH3NPW3OLJNDDZ67U3PDUYFKKDF6WB74EBRONATPQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.662919455 (501h21m43.53400249s) [noise=true , address=77.90.31.18:4822 , piece_id=BEBBCJKIZXWEPGKWT4CKULV4SUZQIS5YZVO4U7FSMICKI7KA42QQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.687034162 (501h21m43.558117207s) [noise=true , address=77.90.31.11:4566 , piece_id=VBI6QVPQTFN4JX7FBFNYHL4OYHBX4MPTIQDQGHBFICF5SBZ45XPQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.767762384 (501h21m43.638845429s) [noise=true , address=77.90.31.15:4702 , piece_id=J3U7MZX4Y4QVWWDLEZL73RTSV3PY3KJJM5V22LIJ5KW3GNF3A4WQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.769485905 (501h21m43.64056894s) [noise=true , address=77.90.31.17:4777 , piece_id=M5T57WEHR6GEA2K4LUKPHTVNJXOB4N44OQQYSPUZBYBSZNN4LUGA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.882129930 (501h21m43.753212975s) [noise=true , address=77.90.31.13:4645 , piece_id=H77LB7OQOYZWJ5TM6FOXIYZEYPH5XKTAVEIE7DXNMHQM3EAB4G3Q, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.882154297 (501h21m43.753237332s) [seg_pos={0 0}, upload=440005]: Segment reached optimal threshold of 14 pieces. Dec 26 12:50:43.882433248 (501h21m43.753516283s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 26 12:50:43.882442595 (501h21m43.75352565s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882668485 (501h21m43.75375153s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection Dec 26 12:50:43.882679026 (501h21m43.753762061s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882691860 (501h21m43.753774905s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj): protocol: expected piece hash; context canceled; EOF Dec 26 12:50:43.882699975 (501h21m43.75378302s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882859289 (501h21m43.753942334s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121bMz2UYi9i53u7ss3SULotcCmAQVhXbNPFr4Hk7XGdgHeVQMw): protocol: write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection; write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection Dec 26 12:50:43.882867174 (501h21m43.753950219s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882886611 (501h21m43.753969646s) [seg_pos={0 0}, upload=440005]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 12:50:43.882980860 (501h21m43.754063895s) [upload=440005]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:50:43.882987884 (501h21m43.754070929s) [upload=440005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:50:44.189484256 (501h21m44.060567291s) [upload=440005]: Done uploading segments... Dec 26 12:50:44.189501439 (501h21m44.060584474s) [upload=440005]: Upload finished. err: <nil> Duration: 55.788726758s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [.....X.......XXX...X.....X.......X.XX......XXX..XXXXXXX.XXXX] duration: 5.00134596s trace: https://jaeger.dev.storj.io/trace/f45800a7d94cace FAIL (2025-12-26 12:52:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001365898s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X..X...X..XX.....XXXX.....XXX....XX....X..X......X.X..XX....] duration: 5.710693182s trace: https://jaeger.dev.storj.io/trace/a7946c4655f3019 FAIL (2025-12-26 12:52:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: 67B4CJKGQOCJNRJK3MKI3T6B4I6A6BFRIS4ZIU5LOATZWTZ5O3TA) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: KAGOUHT7WVAZFM3Y4RIAWDVDHF4WB3TS2IGMX2OU74B4P4B3ZXRA) context deadline exceeded ======LOGS====== Dec 26 12:49:02.447043594 (501h20m2.318126639s) [upload=439980]: Starting upload Dec 26 12:49:02.447156158 (501h20m2.318239203s) [upload=439980]: Done starting upload Dec 26 12:49:02.447204801 (501h20m2.318287846s) [upload=439980]: Uploading segments... Dec 26 12:49:02.447398079 (501h20m2.318481124s) [upload=439980]: Got next segment. Inline: false Dec 26 12:49:02.447447503 (501h20m2.318530538s) [upload=439980]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:49:02.447454657 (501h20m2.318537702s) [upload=439980]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:49:02.688830688 (501h20m2.559913744s) [seg_pos={0 0}, upload=439980]: Begin upload segment... Dec 26 12:49:02.688890753 (501h20m2.559973788s) [seg_pos={0 0}, upload=439980]: Done begin upload segment. Dec 26 12:49:02.688920770 (501h20m2.560003815s) [noise=true , address=145.79.227.45:4229 , piece_id=TRG4WAIVFL33BVBJ5PCIQ5EUPZ4LI5UL2TJLK6EN5HMVLD5J5EMQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.689093059 (501h20m2.560176094s) [noise=true , address=145.79.227.46:4602 , piece_id=CPIT2K2YAAC55GDEC5LPCNJ22SSII7MVTBUORL6X2HACU2CKOABQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.689721966 (501h20m2.560805021s) [noise=true , address=145.79.227.31:4799 , piece_id=SE75ODCMXKIUAR3JTTYRYSHF6BOWH6WMCXGRX4KWRHZPLFJDI57A, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.690107059 (501h20m2.561190104s) [noise=true , address=145.79.227.15:4876 , piece_id=2UCA4YCUUVYCOWZ4HZF3R2T7GT64Z5KZKL6N6QXVMZR26MVGDR3A, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.690704497 (501h20m2.561787542s) [noise=true , address=145.79.227.41:4087 , piece_id=L77EWW6ZI4P5KXRBD7D4WQKMZ3J5YO42LUI53FBIAOQRT4D2X5XQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.691216792 (501h20m2.562299838s) [noise=true , address=145.79.227.17:4971 , piece_id=ZEELPS5ZQTMLZ2LIPNUJ4XRV7TID4VOYPJXVUOL3DA5R3LMPU2VQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.691700504 (501h20m2.562783549s) [noise=true , address=45.91.116.14:4191 , piece_id=ANVMW7PZB54QZQCCIJTFZAMCLNHQP6M5B7SO2XDG72RNE2P5DA2Q, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.692170979 (501h20m2.563254024s) [noise=true , address=145.79.227.35:4481 , piece_id=HM2ZHXJJTEELPP76YNF3FV3YKQ4X64QJPOXDIPANOICYBJZNEW5Q, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.692728280 (501h20m2.563811325s) [noise=true , address=145.79.227.26:4654 , piece_id=QLPH4VGYQJGALFYUZOJ23OWOMOYNPIQ3MSYM6XQIA5ZFYTQ2RAWQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.694872544 (501h20m2.565955589s) [noise=true , address=145.79.227.11:4731 , piece_id=N6N2IEWXNY3B4ASST2B2EZ5375T2F6AV6JXELAYSRW4X2MCCGRTQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.697026275 (501h20m2.56810931s) [noise=true , address=45.91.116.11:4117 , piece_id=IENGJEVLVYZNAOBXBKFLAEJ3EOBYYSL7C3LGDYXYD52TGNR5R6EQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.699009491 (501h20m2.570092536s) [noise=true , address=77.90.31.16:4735 , piece_id=KAGOUHT7WVAZFM3Y4RIAWDVDHF4WB3TS2IGMX2OU74B4P4B3ZXRA, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.700818746 (501h20m2.571901791s) [noise=true , address=145.79.227.39:4014 , piece_id=VGHWBYL4OHJBVBCBPHL7DNY5O5C57OHL766P4JZEWRKPSBOEOONQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.703072998 (501h20m2.574156033s) [noise=true , address=77.90.31.10:4519 , piece_id=QCIWEIL43J22JV6ESHU2TPHBGBDPS2CMWUSVRKW4HL5XCETPQPCA, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.707037057 (501h20m2.578120102s) [noise=true , address=145.79.227.36:4518 , piece_id=URZLNXCY4VBUY4HSBWP6WKCM2RUMEGIL4UT7ZZM4QA6BJLEF7LAQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.708645239 (501h20m2.579728294s) [noise=true , address=77.90.31.14:4663 , piece_id=67B4CJKGQOCJNRJK3MKI3T6B4I6A6BFRIS4ZIU5LOATZWTZ5O3TA, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:04.416018446 (501h20m4.287101491s) [upload=439980]: Next returned nil segment Dec 26 12:49:04.416077238 (501h20m4.287160283s) [upload=439980]: Waiting for error group managing segments... Dec 26 12:49:04.703930962 (501h20m4.575014007s) [noise=true , address=145.79.227.15:4876 , piece_id=2UCA4YCUUVYCOWZ4HZF3R2T7GT64Z5KZKL6N6QXVMZR26MVGDR3A, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.704247905 (501h20m4.57533096s) [noise=true , address=145.79.227.35:4481 , piece_id=HM2ZHXJJTEELPP76YNF3FV3YKQ4X64QJPOXDIPANOICYBJZNEW5Q, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.707534473 (501h20m4.578617508s) [noise=true , address=145.79.227.26:4654 , piece_id=QLPH4VGYQJGALFYUZOJ23OWOMOYNPIQ3MSYM6XQIA5ZFYTQ2RAWQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.707569971 (501h20m4.578653006s) [noise=true , address=145.79.227.11:4731 , piece_id=N6N2IEWXNY3B4ASST2B2EZ5375T2F6AV6JXELAYSRW4X2MCCGRTQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.713069783 (501h20m4.584152828s) [noise=true , address=145.79.227.17:4971 , piece_id=ZEELPS5ZQTMLZ2LIPNUJ4XRV7TID4VOYPJXVUOL3DA5R3LMPU2VQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.715337060 (501h20m4.586420106s) [noise=true , address=145.79.227.46:4602 , piece_id=CPIT2K2YAAC55GDEC5LPCNJ22SSII7MVTBUORL6X2HACU2CKOABQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.719134893 (501h20m4.590217938s) [noise=true , address=145.79.227.45:4229 , piece_id=TRG4WAIVFL33BVBJ5PCIQ5EUPZ4LI5UL2TJLK6EN5HMVLD5J5EMQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.721251483 (501h20m4.592334518s) [noise=true , address=145.79.227.36:4518 , piece_id=URZLNXCY4VBUY4HSBWP6WKCM2RUMEGIL4UT7ZZM4QA6BJLEF7LAQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.721582844 (501h20m4.592665889s) [noise=true , address=145.79.227.41:4087 , piece_id=L77EWW6ZI4P5KXRBD7D4WQKMZ3J5YO42LUI53FBIAOQRT4D2X5XQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.728346563 (501h20m4.599429608s) [noise=true , address=145.79.227.31:4799 , piece_id=SE75ODCMXKIUAR3JTTYRYSHF6BOWH6WMCXGRX4KWRHZPLFJDI57A, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.733132587 (501h20m4.604215632s) [noise=true , address=145.79.227.39:4014 , piece_id=VGHWBYL4OHJBVBCBPHL7DNY5O5C57OHL766P4JZEWRKPSBOEOONQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.978493166 (501h20m4.849576202s) [noise=true , address=77.90.31.16:4735 , piece_id=KAGOUHT7WVAZFM3Y4RIAWDVDHF4WB3TS2IGMX2OU74B4P4B3ZXRA, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.978960205 (501h20m4.85004326s) [noise=true , address=77.90.31.14:4663 , piece_id=67B4CJKGQOCJNRJK3MKI3T6B4I6A6BFRIS4ZIU5LOATZWTZ5O3TA, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.978982949 (501h20m4.850065994s) [seg_pos={0 0}, upload=439980]: Segment reached optimal threshold of 13 pieces. Dec 26 12:49:04.979234939 (501h20m4.850317984s) [noise=true , address=45.91.116.11:4117 , piece_id=IENGJEVLVYZNAOBXBKFLAEJ3EOBYYSL7C3LGDYXYD52TGNR5R6EQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 26 12:49:04.979244006 (501h20m4.850327051s) [noise=true , address=45.91.116.11:4117 , piece_id=IENGJEVLVYZNAOBXBKFLAEJ3EOBYYSL7C3LGDYXYD52TGNR5R6EQ, seg_pos={0 0}, upload=439980]: Long tail cancellation due to optimal threshold Dec 26 12:49:04.979607709 (501h20m4.850690754s) [noise=true , address=77.90.31.10:4519 , piece_id=QCIWEIL43J22JV6ESHU2TPHBGBDPS2CMWUSVRKW4HL5XCETPQPCA, seg_pos={0 0}, upload=439980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 26 12:49:04.979615263 (501h20m4.850698298s) [noise=true , address=77.90.31.10:4519 , piece_id=QCIWEIL43J22JV6ESHU2TPHBGBDPS2CMWUSVRKW4HL5XCETPQPCA, seg_pos={0 0}, upload=439980]: Long tail cancellation due to optimal threshold Dec 26 12:49:04.979736964 (501h20m4.850820009s) [noise=true , address=45.91.116.14:4191 , piece_id=ANVMW7PZB54QZQCCIJTFZAMCLNHQP6M5B7SO2XDG72RNE2P5DA2Q, seg_pos={0 0}, upload=439980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:59208->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:59208->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59208->45.91.116.14:4191: use of closed network connection Dec 26 12:49:04.979748377 (501h20m4.850831412s) [noise=true , address=45.91.116.14:4191 , piece_id=ANVMW7PZB54QZQCCIJTFZAMCLNHQP6M5B7SO2XDG72RNE2P5DA2Q, seg_pos={0 0}, upload=439980]: Long tail cancellation due to optimal threshold Dec 26 12:49:04.979769035 (501h20m4.85085207s) [seg_pos={0 0}, upload=439980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:49:04.979846002 (501h20m4.850929047s) [upload=439980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:49:04.979851082 (501h20m4.850934117s) [upload=439980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:49:05.271932757 (501h20m5.143015792s) [upload=439980]: Done uploading segments... Dec 26 12:49:05.271960820 (501h20m5.143043855s) [upload=439980]: Upload finished. err: <nil> Duration: 5.710738257s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [......X.......XXX...X.....X.......X.XX......XXX..XXXXXXX.XXX] duration: 5.002336555s trace: https://jaeger.dev.storj.io/trace/598a6fad3a038a16 FAIL (2025-12-26 12:52:01 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002363275s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 7 out of 60 is failed (threshold=3) [...........................X....................X.XXX...X..X] duration: 1m0.003064344s trace: https://jaeger.dev.storj.io/trace/4fdd8919ff1f03ab FAIL (2025-12-26 12:51:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 64.143:4695 , piece_id=MJU7BWFRV3XEUAUXWLV3PILJFJ3EEYU5JQDR7ELFP56K7UMOJ5SQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.504565702 (501h21m41.375648747s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.504919806 (501h21m41.376002841s) [noise=true , address=77.90.31.13:4645 , piece_id=H77LB7OQOYZWJ5TM6FOXIYZEYPH5XKTAVEIE7DXNMHQM3EAB4G3Q, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.505319897 (501h21m41.376402942s) [noise=true , address=77.90.31.18:4822 , piece_id=BEBBCJKIZXWEPGKWT4CKULV4SUZQIS5YZVO4U7FSMICKI7KA42QQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.505772810 (501h21m41.376855855s) [noise=true , address=77.90.31.11:4566 , piece_id=VBI6QVPQTFN4JX7FBFNYHL4OYHBX4MPTIQDQGHBFICF5SBZ45XPQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.506212748 (501h21m41.377295793s) [noise=true , address=145.79.227.40:4081 , piece_id=TLYGZBCLBK5QYAX2O3CAEPQQW6GHSNOCWVMCJK64PUF2ZVB3QZOQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.506727949 (501h21m41.377810994s) [noise=true , address=77.90.31.15:4702 , piece_id=J3U7MZX4Y4QVWWDLEZL73RTSV3PY3KJJM5V22LIJ5KW3GNF3A4WQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.508679004 (501h21m41.379762039s) [noise=true , address=77.90.31.17:4777 , piece_id=M5T57WEHR6GEA2K4LUKPHTVNJXOB4N44OQQYSPUZBYBSZNN4LUGA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.510244746 (501h21m41.381327791s) [noise=true , address=45.91.116.8:4062 , piece_id=ZIQQ76IAON3KH3NPW3OLJNDDZ67U3PDUYFKKDF6WB74EBRONATPQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.513404332 (501h21m41.384487367s) [noise=true , address=64.49.13.138:4481 , piece_id=AYNG7FPLCAWWPCQOREPGIKVMHYRW7UHQC6W2M6GD3SYSKNQZB2FA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.515190553 (501h21m41.386273588s) [noise=true , address=145.79.227.17:4002 , piece_id=L73THH6HKT6X2U3UVHRF56DZJXRN56XEQBYCEQRMK6XPZTH5G5BQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.516689447 (501h21m41.387772482s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.518014290 (501h21m41.389097335s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.519302834 (501h21m41.390385879s) [noise=true , address=145.79.227.34:4918 , piece_id=NQSH2PFCL4NXVGTYWT2CSBIV3TQSEEBDUIPQBI33ZJVHVP7DWHIQ, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:41.520341321 (501h21m41.391424366s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Uploading piece... Dec 26 12:50:42.797832918 (501h21m42.668915953s) [upload=440005]: Next returned nil segment Dec 26 12:50:42.797878615 (501h21m42.66896167s) [upload=440005]: Waiting for error group managing segments... Dec 26 12:50:43.150930207 (501h21m43.022013273s) [noise=true , address=64.49.13.138:4481 , piece_id=AYNG7FPLCAWWPCQOREPGIKVMHYRW7UHQC6W2M6GD3SYSKNQZB2FA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.162932632 (501h21m43.034015667s) [noise=true , address=193.149.164.143:4695 , piece_id=MJU7BWFRV3XEUAUXWLV3PILJFJ3EEYU5JQDR7ELFP56K7UMOJ5SQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.374778985 (501h21m43.24586202s) [noise=true , address=145.79.227.40:4081 , piece_id=TLYGZBCLBK5QYAX2O3CAEPQQW6GHSNOCWVMCJK64PUF2ZVB3QZOQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.375783878 (501h21m43.246866913s) [noise=true , address=145.79.227.23:4580 , piece_id=YANWHT6JCLD6JNE7EFJFRZXHNKNKRTUXHEU2NHQLK5U4FNB7JR7A, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.391967825 (501h21m43.263050871s) [noise=true , address=145.79.227.34:4918 , piece_id=NQSH2PFCL4NXVGTYWT2CSBIV3TQSEEBDUIPQBI33ZJVHVP7DWHIQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.392038590 (501h21m43.263121625s) [noise=true , address=145.79.227.52:4476 , piece_id=B7F5LMSUB26MRAIRWA2U5J5IFMZGXDDJUZZKPA6PFJ6WQQ5QGCCQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.405638596 (501h21m43.276721641s) [noise=true , address=145.79.227.17:4002 , piece_id=L73THH6HKT6X2U3UVHRF56DZJXRN56XEQBYCEQRMK6XPZTH5G5BQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.599725422 (501h21m43.470808477s) [noise=true , address=45.91.116.6:4002 , piece_id=UYE5IA3ZW7ICL2F7ALYTXFDMR53CAXNHBKIK34JK6IBQCJJRF3AA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.638993344 (501h21m43.510076389s) [noise=true , address=45.91.116.8:4062 , piece_id=ZIQQ76IAON3KH3NPW3OLJNDDZ67U3PDUYFKKDF6WB74EBRONATPQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.662919455 (501h21m43.53400249s) [noise=true , address=77.90.31.18:4822 , piece_id=BEBBCJKIZXWEPGKWT4CKULV4SUZQIS5YZVO4U7FSMICKI7KA42QQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.687034162 (501h21m43.558117207s) [noise=true , address=77.90.31.11:4566 , piece_id=VBI6QVPQTFN4JX7FBFNYHL4OYHBX4MPTIQDQGHBFICF5SBZ45XPQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.767762384 (501h21m43.638845429s) [noise=true , address=77.90.31.15:4702 , piece_id=J3U7MZX4Y4QVWWDLEZL73RTSV3PY3KJJM5V22LIJ5KW3GNF3A4WQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.769485905 (501h21m43.64056894s) [noise=true , address=77.90.31.17:4777 , piece_id=M5T57WEHR6GEA2K4LUKPHTVNJXOB4N44OQQYSPUZBYBSZNN4LUGA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.882129930 (501h21m43.753212975s) [noise=true , address=77.90.31.13:4645 , piece_id=H77LB7OQOYZWJ5TM6FOXIYZEYPH5XKTAVEIE7DXNMHQM3EAB4G3Q, seg_pos={0 0}, upload=440005]: Done uploading piece. err: <nil> Dec 26 12:50:43.882154297 (501h21m43.753237332s) [seg_pos={0 0}, upload=440005]: Segment reached optimal threshold of 14 pieces. Dec 26 12:50:43.882433248 (501h21m43.753516283s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1oUUQD6GJU2WHCYnyp98du4BG8wAXMvsoLvvaFfuAsfqA6mCfb): protocol: expected piece hash; context canceled; EOF Dec 26 12:50:43.882442595 (501h21m43.75352565s) [noise=true , address=145.79.227.30:4774 , piece_id=SIO5G2VLMZ2SYO3ANMMYYNNSVPAEUDO4HC4W73PEAKR2GHH45VAA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882668485 (501h21m43.75375153s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57778->64.49.13.169:4710: use of closed network connection Dec 26 12:50:43.882679026 (501h21m43.753762061s) [noise=true , address=64.49.13.169:4710 , piece_id=HUHQIJN6FWQQDHGNX36XOPUBWP3D2EKX4HXYCNDRAZNAJ2MX3PLQ, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882691860 (501h21m43.753774905s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj): protocol: expected piece hash; context canceled; EOF Dec 26 12:50:43.882699975 (501h21m43.75378302s) [noise=true , address=77.90.31.12:4595 , piece_id=VWGAHFSJSCY6MW5WDLFOTSYKJQMHAPN3ZTM7JPW675FP3TQNKWBA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882859289 (501h21m43.753942334s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121bMz2UYi9i53u7ss3SULotcCmAQVhXbNPFr4Hk7XGdgHeVQMw): protocol: write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection; write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33800->45.91.116.7:4013: use of closed network connection Dec 26 12:50:43.882867174 (501h21m43.753950219s) [noise=true , address=45.91.116.7:4013 , piece_id=XNVKHSEFSW67EEPYWJ3DUCM2XFCXBCKMJHLRFLYG2B565UXCDNEA, seg_pos={0 0}, upload=440005]: Long tail cancellation due to optimal threshold Dec 26 12:50:43.882886611 (501h21m43.753969646s) [seg_pos={0 0}, upload=440005]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 12:50:43.882980860 (501h21m43.754063895s) [upload=440005]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:50:43.882987884 (501h21m43.754070929s) [upload=440005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:50:44.189484256 (501h21m44.060567291s) [upload=440005]: Done uploading segments... Dec 26 12:50:44.189501439 (501h21m44.060584474s) [upload=440005]: Upload finished. err: <nil> Duration: 1m0.003088571s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X..X...X..XX.....XXXX.....XXX....XX....X..X......X.X..XX...] duration: 5.75897663s trace: https://jaeger.dev.storj.io/trace/a7946c4655f3019 FAIL (2025-12-26 12:51:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: 67B4CJKGQOCJNRJK3MKI3T6B4I6A6BFRIS4ZIU5LOATZWTZ5O3TA) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: KAGOUHT7WVAZFM3Y4RIAWDVDHF4WB3TS2IGMX2OU74B4P4B3ZXRA) context deadline exceeded ======LOGS====== Dec 26 12:49:02.447043594 (501h20m2.318126639s) [upload=439980]: Starting upload Dec 26 12:49:02.447156158 (501h20m2.318239203s) [upload=439980]: Done starting upload Dec 26 12:49:02.447204801 (501h20m2.318287846s) [upload=439980]: Uploading segments... Dec 26 12:49:02.447398079 (501h20m2.318481124s) [upload=439980]: Got next segment. Inline: false Dec 26 12:49:02.447447503 (501h20m2.318530538s) [upload=439980]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:49:02.447454657 (501h20m2.318537702s) [upload=439980]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:49:02.688830688 (501h20m2.559913744s) [seg_pos={0 0}, upload=439980]: Begin upload segment... Dec 26 12:49:02.688890753 (501h20m2.559973788s) [seg_pos={0 0}, upload=439980]: Done begin upload segment. Dec 26 12:49:02.688920770 (501h20m2.560003815s) [noise=true , address=145.79.227.45:4229 , piece_id=TRG4WAIVFL33BVBJ5PCIQ5EUPZ4LI5UL2TJLK6EN5HMVLD5J5EMQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.689093059 (501h20m2.560176094s) [noise=true , address=145.79.227.46:4602 , piece_id=CPIT2K2YAAC55GDEC5LPCNJ22SSII7MVTBUORL6X2HACU2CKOABQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.689721966 (501h20m2.560805021s) [noise=true , address=145.79.227.31:4799 , piece_id=SE75ODCMXKIUAR3JTTYRYSHF6BOWH6WMCXGRX4KWRHZPLFJDI57A, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.690107059 (501h20m2.561190104s) [noise=true , address=145.79.227.15:4876 , piece_id=2UCA4YCUUVYCOWZ4HZF3R2T7GT64Z5KZKL6N6QXVMZR26MVGDR3A, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.690704497 (501h20m2.561787542s) [noise=true , address=145.79.227.41:4087 , piece_id=L77EWW6ZI4P5KXRBD7D4WQKMZ3J5YO42LUI53FBIAOQRT4D2X5XQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.691216792 (501h20m2.562299838s) [noise=true , address=145.79.227.17:4971 , piece_id=ZEELPS5ZQTMLZ2LIPNUJ4XRV7TID4VOYPJXVUOL3DA5R3LMPU2VQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.691700504 (501h20m2.562783549s) [noise=true , address=45.91.116.14:4191 , piece_id=ANVMW7PZB54QZQCCIJTFZAMCLNHQP6M5B7SO2XDG72RNE2P5DA2Q, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.692170979 (501h20m2.563254024s) [noise=true , address=145.79.227.35:4481 , piece_id=HM2ZHXJJTEELPP76YNF3FV3YKQ4X64QJPOXDIPANOICYBJZNEW5Q, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.692728280 (501h20m2.563811325s) [noise=true , address=145.79.227.26:4654 , piece_id=QLPH4VGYQJGALFYUZOJ23OWOMOYNPIQ3MSYM6XQIA5ZFYTQ2RAWQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.694872544 (501h20m2.565955589s) [noise=true , address=145.79.227.11:4731 , piece_id=N6N2IEWXNY3B4ASST2B2EZ5375T2F6AV6JXELAYSRW4X2MCCGRTQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.697026275 (501h20m2.56810931s) [noise=true , address=45.91.116.11:4117 , piece_id=IENGJEVLVYZNAOBXBKFLAEJ3EOBYYSL7C3LGDYXYD52TGNR5R6EQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.699009491 (501h20m2.570092536s) [noise=true , address=77.90.31.16:4735 , piece_id=KAGOUHT7WVAZFM3Y4RIAWDVDHF4WB3TS2IGMX2OU74B4P4B3ZXRA, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.700818746 (501h20m2.571901791s) [noise=true , address=145.79.227.39:4014 , piece_id=VGHWBYL4OHJBVBCBPHL7DNY5O5C57OHL766P4JZEWRKPSBOEOONQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.703072998 (501h20m2.574156033s) [noise=true , address=77.90.31.10:4519 , piece_id=QCIWEIL43J22JV6ESHU2TPHBGBDPS2CMWUSVRKW4HL5XCETPQPCA, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.707037057 (501h20m2.578120102s) [noise=true , address=145.79.227.36:4518 , piece_id=URZLNXCY4VBUY4HSBWP6WKCM2RUMEGIL4UT7ZZM4QA6BJLEF7LAQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.708645239 (501h20m2.579728294s) [noise=true , address=77.90.31.14:4663 , piece_id=67B4CJKGQOCJNRJK3MKI3T6B4I6A6BFRIS4ZIU5LOATZWTZ5O3TA, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:04.416018446 (501h20m4.287101491s) [upload=439980]: Next returned nil segment Dec 26 12:49:04.416077238 (501h20m4.287160283s) [upload=439980]: Waiting for error group managing segments... Dec 26 12:49:04.703930962 (501h20m4.575014007s) [noise=true , address=145.79.227.15:4876 , piece_id=2UCA4YCUUVYCOWZ4HZF3R2T7GT64Z5KZKL6N6QXVMZR26MVGDR3A, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.704247905 (501h20m4.57533096s) [noise=true , address=145.79.227.35:4481 , piece_id=HM2ZHXJJTEELPP76YNF3FV3YKQ4X64QJPOXDIPANOICYBJZNEW5Q, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.707534473 (501h20m4.578617508s) [noise=true , address=145.79.227.26:4654 , piece_id=QLPH4VGYQJGALFYUZOJ23OWOMOYNPIQ3MSYM6XQIA5ZFYTQ2RAWQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.707569971 (501h20m4.578653006s) [noise=true , address=145.79.227.11:4731 , piece_id=N6N2IEWXNY3B4ASST2B2EZ5375T2F6AV6JXELAYSRW4X2MCCGRTQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.713069783 (501h20m4.584152828s) [noise=true , address=145.79.227.17:4971 , piece_id=ZEELPS5ZQTMLZ2LIPNUJ4XRV7TID4VOYPJXVUOL3DA5R3LMPU2VQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.715337060 (501h20m4.586420106s) [noise=true , address=145.79.227.46:4602 , piece_id=CPIT2K2YAAC55GDEC5LPCNJ22SSII7MVTBUORL6X2HACU2CKOABQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.719134893 (501h20m4.590217938s) [noise=true , address=145.79.227.45:4229 , piece_id=TRG4WAIVFL33BVBJ5PCIQ5EUPZ4LI5UL2TJLK6EN5HMVLD5J5EMQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.721251483 (501h20m4.592334518s) [noise=true , address=145.79.227.36:4518 , piece_id=URZLNXCY4VBUY4HSBWP6WKCM2RUMEGIL4UT7ZZM4QA6BJLEF7LAQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.721582844 (501h20m4.592665889s) [noise=true , address=145.79.227.41:4087 , piece_id=L77EWW6ZI4P5KXRBD7D4WQKMZ3J5YO42LUI53FBIAOQRT4D2X5XQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.728346563 (501h20m4.599429608s) [noise=true , address=145.79.227.31:4799 , piece_id=SE75ODCMXKIUAR3JTTYRYSHF6BOWH6WMCXGRX4KWRHZPLFJDI57A, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.733132587 (501h20m4.604215632s) [noise=true , address=145.79.227.39:4014 , piece_id=VGHWBYL4OHJBVBCBPHL7DNY5O5C57OHL766P4JZEWRKPSBOEOONQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.978493166 (501h20m4.849576202s) [noise=true , address=77.90.31.16:4735 , piece_id=KAGOUHT7WVAZFM3Y4RIAWDVDHF4WB3TS2IGMX2OU74B4P4B3ZXRA, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.978960205 (501h20m4.85004326s) [noise=true , address=77.90.31.14:4663 , piece_id=67B4CJKGQOCJNRJK3MKI3T6B4I6A6BFRIS4ZIU5LOATZWTZ5O3TA, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.978982949 (501h20m4.850065994s) [seg_pos={0 0}, upload=439980]: Segment reached optimal threshold of 13 pieces. Dec 26 12:49:04.979234939 (501h20m4.850317984s) [noise=true , address=45.91.116.11:4117 , piece_id=IENGJEVLVYZNAOBXBKFLAEJ3EOBYYSL7C3LGDYXYD52TGNR5R6EQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 26 12:49:04.979244006 (501h20m4.850327051s) [noise=true , address=45.91.116.11:4117 , piece_id=IENGJEVLVYZNAOBXBKFLAEJ3EOBYYSL7C3LGDYXYD52TGNR5R6EQ, seg_pos={0 0}, upload=439980]: Long tail cancellation due to optimal threshold Dec 26 12:49:04.979607709 (501h20m4.850690754s) [noise=true , address=77.90.31.10:4519 , piece_id=QCIWEIL43J22JV6ESHU2TPHBGBDPS2CMWUSVRKW4HL5XCETPQPCA, seg_pos={0 0}, upload=439980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 26 12:49:04.979615263 (501h20m4.850698298s) [noise=true , address=77.90.31.10:4519 , piece_id=QCIWEIL43J22JV6ESHU2TPHBGBDPS2CMWUSVRKW4HL5XCETPQPCA, seg_pos={0 0}, upload=439980]: Long tail cancellation due to optimal threshold Dec 26 12:49:04.979736964 (501h20m4.850820009s) [noise=true , address=45.91.116.14:4191 , piece_id=ANVMW7PZB54QZQCCIJTFZAMCLNHQP6M5B7SO2XDG72RNE2P5DA2Q, seg_pos={0 0}, upload=439980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:59208->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:59208->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59208->45.91.116.14:4191: use of closed network connection Dec 26 12:49:04.979748377 (501h20m4.850831412s) [noise=true , address=45.91.116.14:4191 , piece_id=ANVMW7PZB54QZQCCIJTFZAMCLNHQP6M5B7SO2XDG72RNE2P5DA2Q, seg_pos={0 0}, upload=439980]: Long tail cancellation due to optimal threshold Dec 26 12:49:04.979769035 (501h20m4.85085207s) [seg_pos={0 0}, upload=439980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:49:04.979846002 (501h20m4.850929047s) [upload=439980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:49:04.979851082 (501h20m4.850934117s) [upload=439980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:49:05.271932757 (501h20m5.143015792s) [upload=439980]: Done uploading segments... Dec 26 12:49:05.271960820 (501h20m5.143043855s) [upload=439980]: Upload finished. err: <nil> Duration: 5.759006296s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [X......X.......XXX...X.....X.......X.XX......XXX..XXXXXXX.XX] duration: 5.001232943s trace: https://jaeger.dev.storj.io/trace/319d2c96b51928b6 FAIL (2025-12-26 12:51:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001257019s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [.X......X.......XXX...X.....X.......X.XX......XXX..XXXXXXX.X] duration: 5.001428356s trace: https://jaeger.dev.storj.io/trace/60759934fa4193c7 FAIL (2025-12-26 12:51:05 UTC) orbiter warning: context deadline exceeded Duration: 5.001447492s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [..X......X.......XXX...X.....X.......X.XX......XXX..XXXXXXX.] duration: 5.000442548s trace: https://jaeger.dev.storj.io/trace/364fd33ca62f4be4 FAIL (2025-12-26 12:50:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000465381s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [..X..X...X..XX.....XXXX.....XXX....XX....X..X......X.X..XX..] duration: 6.260725515s trace: https://jaeger.dev.storj.io/trace/a7946c4655f3019 FAIL (2025-12-26 12:50:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: 67B4CJKGQOCJNRJK3MKI3T6B4I6A6BFRIS4ZIU5LOATZWTZ5O3TA) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: KAGOUHT7WVAZFM3Y4RIAWDVDHF4WB3TS2IGMX2OU74B4P4B3ZXRA) context deadline exceeded ======LOGS====== Dec 26 12:49:02.447043594 (501h20m2.318126639s) [upload=439980]: Starting upload Dec 26 12:49:02.447156158 (501h20m2.318239203s) [upload=439980]: Done starting upload Dec 26 12:49:02.447204801 (501h20m2.318287846s) [upload=439980]: Uploading segments... Dec 26 12:49:02.447398079 (501h20m2.318481124s) [upload=439980]: Got next segment. Inline: false Dec 26 12:49:02.447447503 (501h20m2.318530538s) [upload=439980]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:49:02.447454657 (501h20m2.318537702s) [upload=439980]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:49:02.688830688 (501h20m2.559913744s) [seg_pos={0 0}, upload=439980]: Begin upload segment... Dec 26 12:49:02.688890753 (501h20m2.559973788s) [seg_pos={0 0}, upload=439980]: Done begin upload segment. Dec 26 12:49:02.688920770 (501h20m2.560003815s) [noise=true , address=145.79.227.45:4229 , piece_id=TRG4WAIVFL33BVBJ5PCIQ5EUPZ4LI5UL2TJLK6EN5HMVLD5J5EMQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.689093059 (501h20m2.560176094s) [noise=true , address=145.79.227.46:4602 , piece_id=CPIT2K2YAAC55GDEC5LPCNJ22SSII7MVTBUORL6X2HACU2CKOABQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.689721966 (501h20m2.560805021s) [noise=true , address=145.79.227.31:4799 , piece_id=SE75ODCMXKIUAR3JTTYRYSHF6BOWH6WMCXGRX4KWRHZPLFJDI57A, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.690107059 (501h20m2.561190104s) [noise=true , address=145.79.227.15:4876 , piece_id=2UCA4YCUUVYCOWZ4HZF3R2T7GT64Z5KZKL6N6QXVMZR26MVGDR3A, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.690704497 (501h20m2.561787542s) [noise=true , address=145.79.227.41:4087 , piece_id=L77EWW6ZI4P5KXRBD7D4WQKMZ3J5YO42LUI53FBIAOQRT4D2X5XQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.691216792 (501h20m2.562299838s) [noise=true , address=145.79.227.17:4971 , piece_id=ZEELPS5ZQTMLZ2LIPNUJ4XRV7TID4VOYPJXVUOL3DA5R3LMPU2VQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.691700504 (501h20m2.562783549s) [noise=true , address=45.91.116.14:4191 , piece_id=ANVMW7PZB54QZQCCIJTFZAMCLNHQP6M5B7SO2XDG72RNE2P5DA2Q, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.692170979 (501h20m2.563254024s) [noise=true , address=145.79.227.35:4481 , piece_id=HM2ZHXJJTEELPP76YNF3FV3YKQ4X64QJPOXDIPANOICYBJZNEW5Q, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.692728280 (501h20m2.563811325s) [noise=true , address=145.79.227.26:4654 , piece_id=QLPH4VGYQJGALFYUZOJ23OWOMOYNPIQ3MSYM6XQIA5ZFYTQ2RAWQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.694872544 (501h20m2.565955589s) [noise=true , address=145.79.227.11:4731 , piece_id=N6N2IEWXNY3B4ASST2B2EZ5375T2F6AV6JXELAYSRW4X2MCCGRTQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.697026275 (501h20m2.56810931s) [noise=true , address=45.91.116.11:4117 , piece_id=IENGJEVLVYZNAOBXBKFLAEJ3EOBYYSL7C3LGDYXYD52TGNR5R6EQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.699009491 (501h20m2.570092536s) [noise=true , address=77.90.31.16:4735 , piece_id=KAGOUHT7WVAZFM3Y4RIAWDVDHF4WB3TS2IGMX2OU74B4P4B3ZXRA, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.700818746 (501h20m2.571901791s) [noise=true , address=145.79.227.39:4014 , piece_id=VGHWBYL4OHJBVBCBPHL7DNY5O5C57OHL766P4JZEWRKPSBOEOONQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.703072998 (501h20m2.574156033s) [noise=true , address=77.90.31.10:4519 , piece_id=QCIWEIL43J22JV6ESHU2TPHBGBDPS2CMWUSVRKW4HL5XCETPQPCA, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.707037057 (501h20m2.578120102s) [noise=true , address=145.79.227.36:4518 , piece_id=URZLNXCY4VBUY4HSBWP6WKCM2RUMEGIL4UT7ZZM4QA6BJLEF7LAQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.708645239 (501h20m2.579728294s) [noise=true , address=77.90.31.14:4663 , piece_id=67B4CJKGQOCJNRJK3MKI3T6B4I6A6BFRIS4ZIU5LOATZWTZ5O3TA, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:04.416018446 (501h20m4.287101491s) [upload=439980]: Next returned nil segment Dec 26 12:49:04.416077238 (501h20m4.287160283s) [upload=439980]: Waiting for error group managing segments... Dec 26 12:49:04.703930962 (501h20m4.575014007s) [noise=true , address=145.79.227.15:4876 , piece_id=2UCA4YCUUVYCOWZ4HZF3R2T7GT64Z5KZKL6N6QXVMZR26MVGDR3A, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.704247905 (501h20m4.57533096s) [noise=true , address=145.79.227.35:4481 , piece_id=HM2ZHXJJTEELPP76YNF3FV3YKQ4X64QJPOXDIPANOICYBJZNEW5Q, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.707534473 (501h20m4.578617508s) [noise=true , address=145.79.227.26:4654 , piece_id=QLPH4VGYQJGALFYUZOJ23OWOMOYNPIQ3MSYM6XQIA5ZFYTQ2RAWQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.707569971 (501h20m4.578653006s) [noise=true , address=145.79.227.11:4731 , piece_id=N6N2IEWXNY3B4ASST2B2EZ5375T2F6AV6JXELAYSRW4X2MCCGRTQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.713069783 (501h20m4.584152828s) [noise=true , address=145.79.227.17:4971 , piece_id=ZEELPS5ZQTMLZ2LIPNUJ4XRV7TID4VOYPJXVUOL3DA5R3LMPU2VQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.715337060 (501h20m4.586420106s) [noise=true , address=145.79.227.46:4602 , piece_id=CPIT2K2YAAC55GDEC5LPCNJ22SSII7MVTBUORL6X2HACU2CKOABQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.719134893 (501h20m4.590217938s) [noise=true , address=145.79.227.45:4229 , piece_id=TRG4WAIVFL33BVBJ5PCIQ5EUPZ4LI5UL2TJLK6EN5HMVLD5J5EMQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.721251483 (501h20m4.592334518s) [noise=true , address=145.79.227.36:4518 , piece_id=URZLNXCY4VBUY4HSBWP6WKCM2RUMEGIL4UT7ZZM4QA6BJLEF7LAQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.721582844 (501h20m4.592665889s) [noise=true , address=145.79.227.41:4087 , piece_id=L77EWW6ZI4P5KXRBD7D4WQKMZ3J5YO42LUI53FBIAOQRT4D2X5XQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.728346563 (501h20m4.599429608s) [noise=true , address=145.79.227.31:4799 , piece_id=SE75ODCMXKIUAR3JTTYRYSHF6BOWH6WMCXGRX4KWRHZPLFJDI57A, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.733132587 (501h20m4.604215632s) [noise=true , address=145.79.227.39:4014 , piece_id=VGHWBYL4OHJBVBCBPHL7DNY5O5C57OHL766P4JZEWRKPSBOEOONQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.978493166 (501h20m4.849576202s) [noise=true , address=77.90.31.16:4735 , piece_id=KAGOUHT7WVAZFM3Y4RIAWDVDHF4WB3TS2IGMX2OU74B4P4B3ZXRA, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.978960205 (501h20m4.85004326s) [noise=true , address=77.90.31.14:4663 , piece_id=67B4CJKGQOCJNRJK3MKI3T6B4I6A6BFRIS4ZIU5LOATZWTZ5O3TA, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.978982949 (501h20m4.850065994s) [seg_pos={0 0}, upload=439980]: Segment reached optimal threshold of 13 pieces. Dec 26 12:49:04.979234939 (501h20m4.850317984s) [noise=true , address=45.91.116.11:4117 , piece_id=IENGJEVLVYZNAOBXBKFLAEJ3EOBYYSL7C3LGDYXYD52TGNR5R6EQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 26 12:49:04.979244006 (501h20m4.850327051s) [noise=true , address=45.91.116.11:4117 , piece_id=IENGJEVLVYZNAOBXBKFLAEJ3EOBYYSL7C3LGDYXYD52TGNR5R6EQ, seg_pos={0 0}, upload=439980]: Long tail cancellation due to optimal threshold Dec 26 12:49:04.979607709 (501h20m4.850690754s) [noise=true , address=77.90.31.10:4519 , piece_id=QCIWEIL43J22JV6ESHU2TPHBGBDPS2CMWUSVRKW4HL5XCETPQPCA, seg_pos={0 0}, upload=439980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 26 12:49:04.979615263 (501h20m4.850698298s) [noise=true , address=77.90.31.10:4519 , piece_id=QCIWEIL43J22JV6ESHU2TPHBGBDPS2CMWUSVRKW4HL5XCETPQPCA, seg_pos={0 0}, upload=439980]: Long tail cancellation due to optimal threshold Dec 26 12:49:04.979736964 (501h20m4.850820009s) [noise=true , address=45.91.116.14:4191 , piece_id=ANVMW7PZB54QZQCCIJTFZAMCLNHQP6M5B7SO2XDG72RNE2P5DA2Q, seg_pos={0 0}, upload=439980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:59208->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:59208->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59208->45.91.116.14:4191: use of closed network connection Dec 26 12:49:04.979748377 (501h20m4.850831412s) [noise=true , address=45.91.116.14:4191 , piece_id=ANVMW7PZB54QZQCCIJTFZAMCLNHQP6M5B7SO2XDG72RNE2P5DA2Q, seg_pos={0 0}, upload=439980]: Long tail cancellation due to optimal threshold Dec 26 12:49:04.979769035 (501h20m4.85085207s) [seg_pos={0 0}, upload=439980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:49:04.979846002 (501h20m4.850929047s) [upload=439980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:49:04.979851082 (501h20m4.850934117s) [upload=439980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:49:05.271932757 (501h20m5.143015792s) [upload=439980]: Done uploading segments... Dec 26 12:49:05.271960820 (501h20m5.143043855s) [upload=439980]: Upload finished. err: <nil> Duration: 6.260761415s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [............................X....................X.XXX...X..] duration: 56.445209667s trace: https://jaeger.dev.storj.io/trace/55808baa5e3da7b9 FAIL (2025-12-26 12:50:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gjesgEXSFr1aGyFgASrVRJgEHTne1KichuGqXwMjN1hHBt4kz, Piece ID: QOXGFQE7CP3745MSKKMAYIDMD4CEGI642K2QGF2Q5TYQHYJ5H7YA) context deadline exceeded; piecestore: (Node ID: 12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, Piece ID: PPFXUZ7QDC33NCS6PMM4Z4SWBGFYPGJPBVQYE3FLPONDFHMXLXSQ) context deadline exceeded; piecestore: (Node ID: 1AvSrhj1nHWLBJrw1SKacnjPTi9g4AzjqKvVFoxtzAdJFYAfQt, Piece ID: FQ5HW32FOAV5VHVLXIFPWWG67DVCKE5OSIDSWAEDJ7FH3USONACA) context deadline exceeded ======LOGS====== 1.996987257 (501h18m31.868070292s) [noise=true , address=145.79.227.29:4954 , piece_id=6BH4PB2YY6UMBZHSU7B4MS64PSFHC5SRFCNSYUWHLHKQTUKR7EDQ, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:31.997520121 (501h18m31.868603166s) [noise=true , address=77.90.31.19:4849 , piece_id=BB52EEC5VJKTJ7H722N7SXUMWBWAUAVFMRHMPORK3KC6QY6LMHZQ, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:31.998001308 (501h18m31.869084353s) [noise=true , address=145.79.227.30:4782 , piece_id=KAEZ3CPVWDFBFEEAKZXLFCGJ7XAYYXRQV5TMXZUS2AD6CBX3HPTA, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:31.998213993 (501h18m31.869297028s) [noise=true , address=145.79.227.44:4226 , piece_id=FQY74SGDMFVPM54JVMLG2DATBKKX4UI2JLO5C5YIUFONTAMPTJQQ, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:31.998539562 (501h18m31.869622607s) [noise=true , address=77.90.31.11:4562 , piece_id=4D42T5I4SG3PZMQNR6NYWA2OQYYRCKTYL3PP33SKHZYQSE3WO5NQ, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:31.998740024 (501h18m31.869823059s) [noise=true , address=145.79.227.43:4171 , piece_id=MDGS63GO7OPP6Y2YYCHJOZSZK25X3YTJHXDEOG4RZAF5LZ4IABZA, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:31.998960043 (501h18m31.870043078s) [noise=true , address=145.79.227.40:4069 , piece_id=ISS3ZLARB55VNPWUMFL3WXHGC3NZMKSFWPLVTPWEVZJ7EXUPF5WQ, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:31.999138473 (501h18m31.870221508s) [noise=true , address=45.91.116.6:4993 , piece_id=PKE5HHLE4GKQ34O3I2FONBGICCHP7TCMTLUFQIC4PIXOU5L5MBTQ, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:31.999374642 (501h18m31.870457677s) [noise=true , address=45.91.116.4:4962 , piece_id=P5SBO3YE6QCFXRVAL62G3HC5XJDNJBPA3PMQEYLNB2U7M46IXKCA, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:31.999529438 (501h18m31.870612483s) [noise=true , address=145.79.227.21:4486 , piece_id=V7DWPLGDLWDYE4TJLGIFMTOLDCXEDBTPUYSWYXPFXOLG3KZDKDQA, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:31.999937926 (501h18m31.871020961s) [noise=true , address=145.79.227.34:4939 , piece_id=FAEDI2PHQ543A5OOP6K73ZDWCM3TPTQQBJPFEYOAGW26J7GX5S6Q, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:32.000931908 (501h18m31.872014943s) [noise=true , address=77.90.31.15:4708 , piece_id=PPFXUZ7QDC33NCS6PMM4Z4SWBGFYPGJPBVQYE3FLPONDFHMXLXSQ, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:32.001377096 (501h18m31.872460141s) [noise=true , address=77.90.31.16:4757 , piece_id=QOXGFQE7CP3745MSKKMAYIDMD4CEGI642K2QGF2Q5TYQHYJ5H7YA, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:32.001947612 (501h18m31.873030647s) [noise=true , address=145.79.227.28:4750 , piece_id=SYOPLUXNCR7KCMR7FX6PMICD4SDW7VLAKNWRWVC3455ESX4P4IKQ, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:32.002400785 (501h18m31.87348383s) [noise=true , address=45.91.116.8:4074 , piece_id=YDTDZNKX5LRO3V3UQHWQB64PS7OBRMIZEBIJQ4TWONL5K2KRZVWA, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:32.002666842 (501h18m31.873749877s) [noise=true , address=45.91.116.11:4146 , piece_id=X7UK2OJXVANXFXYCWRZ6M3UIYULU4DFJ4I2UESRR7PRYC6NCXJOA, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:32.003349923 (501h18m31.874432968s) [noise=true , address=77.90.31.14:4671 , piece_id=O7VFBBQLQ5FBP4UPQFELDDKB5JR4GUEFY6I62BVS4D7VCXRT7R5A, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:33.469721230 (501h18m33.340804265s) [upload=439960]: Next returned nil segment Dec 26 12:47:33.469757930 (501h18m33.340840975s) [upload=439960]: Waiting for error group managing segments... Dec 26 12:47:33.864743802 (501h18m33.735826837s) [noise=true , address=145.79.227.29:4954 , piece_id=6BH4PB2YY6UMBZHSU7B4MS64PSFHC5SRFCNSYUWHLHKQTUKR7EDQ, seg_pos={0 0}, upload=439960]: Done uploading piece. err: <nil> Dec 26 12:47:33.878583104 (501h18m33.749666149s) [noise=true , address=145.79.227.21:4486 , piece_id=V7DWPLGDLWDYE4TJLGIFMTOLDCXEDBTPUYSWYXPFXOLG3KZDKDQA, seg_pos={0 0}, upload=439960]: Done uploading piece. err: <nil> Dec 26 12:47:33.885532327 (501h18m33.756615362s) [noise=true , address=145.79.227.28:4750 , piece_id=SYOPLUXNCR7KCMR7FX6PMICD4SDW7VLAKNWRWVC3455ESX4P4IKQ, seg_pos={0 0}, upload=439960]: Done uploading piece. err: <nil> Dec 26 12:47:33.889919020 (501h18m33.761002065s) [noise=true , address=145.79.227.43:4171 , piece_id=MDGS63GO7OPP6Y2YYCHJOZSZK25X3YTJHXDEOG4RZAF5LZ4IABZA, seg_pos={0 0}, upload=439960]: Done uploading piece. err: <nil> Dec 26 12:47:34.047203828 (501h18m33.918286873s) [noise=true , address=77.90.31.15:4708 , piece_id=PPFXUZ7QDC33NCS6PMM4Z4SWBGFYPGJPBVQYE3FLPONDFHMXLXSQ, seg_pos={0 0}, upload=439960]: Done uploading piece. err: <nil> Dec 26 12:47:34.070368728 (501h18m33.941451763s) [noise=true , address=45.91.116.4:4962 , piece_id=P5SBO3YE6QCFXRVAL62G3HC5XJDNJBPA3PMQEYLNB2U7M46IXKCA, seg_pos={0 0}, upload=439960]: Done uploading piece. err: <nil> Dec 26 12:47:34.091662365 (501h18m33.9627454s) [noise=true , address=77.90.31.16:4757 , piece_id=QOXGFQE7CP3745MSKKMAYIDMD4CEGI642K2QGF2Q5TYQHYJ5H7YA, seg_pos={0 0}, upload=439960]: Done uploading piece. err: <nil> Dec 26 12:47:34.143840831 (501h18m34.014923867s) [noise=true , address=77.90.31.19:4849 , piece_id=BB52EEC5VJKTJ7H722N7SXUMWBWAUAVFMRHMPORK3KC6QY6LMHZQ, seg_pos={0 0}, upload=439960]: Done uploading piece. err: <nil> Dec 26 12:47:34.601885600 (501h18m34.472968655s) [noise=true , address=77.90.31.14:4671 , piece_id=O7VFBBQLQ5FBP4UPQFELDDKB5JR4GUEFY6I62BVS4D7VCXRT7R5A, seg_pos={0 0}, upload=439960]: Done uploading piece. err: <nil> Dec 26 12:47:34.776497801 (501h18m34.647580836s) [noise=true , address=145.79.227.40:4069 , piece_id=ISS3ZLARB55VNPWUMFL3WXHGC3NZMKSFWPLVTPWEVZJ7EXUPF5WQ, seg_pos={0 0}, upload=439960]: Done uploading piece. err: <nil> Dec 26 12:47:34.777613185 (501h18m34.64869623s) [noise=true , address=145.79.227.44:4226 , piece_id=FQY74SGDMFVPM54JVMLG2DATBKKX4UI2JLO5C5YIUFONTAMPTJQQ, seg_pos={0 0}, upload=439960]: Done uploading piece. err: <nil> Dec 26 12:47:34.792547793 (501h18m34.663630828s) [noise=true , address=145.79.227.30:4782 , piece_id=KAEZ3CPVWDFBFEEAKZXLFCGJ7XAYYXRQV5TMXZUS2AD6CBX3HPTA, seg_pos={0 0}, upload=439960]: Done uploading piece. err: <nil> Dec 26 12:47:34.830745548 (501h18m34.701828583s) [noise=true , address=77.90.31.10:4531 , piece_id=FQ5HW32FOAV5VHVLXIFPWWG67DVCKE5OSIDSWAEDJ7FH3USONACA, seg_pos={0 0}, upload=439960]: Done uploading piece. err: <nil> Dec 26 12:47:34.951612160 (501h18m34.822695205s) [noise=true , address=145.79.227.34:4939 , piece_id=FAEDI2PHQ543A5OOP6K73ZDWCM3TPTQQBJPFEYOAGW26J7GX5S6Q, seg_pos={0 0}, upload=439960]: Done uploading piece. err: <nil> Dec 26 12:47:34.951634812 (501h18m34.822717857s) [seg_pos={0 0}, upload=439960]: Segment reached optimal threshold of 14 pieces. Dec 26 12:47:34.951856816 (501h18m34.822939851s) [noise=true , address=45.91.116.11:4146 , piece_id=X7UK2OJXVANXFXYCWRZ6M3UIYULU4DFJ4I2UESRR7PRYC6NCXJOA, seg_pos={0 0}, upload=439960]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp): protocol: expected piece hash; context canceled; EOF Dec 26 12:47:34.951864901 (501h18m34.822947946s) [noise=true , address=45.91.116.11:4146 , piece_id=X7UK2OJXVANXFXYCWRZ6M3UIYULU4DFJ4I2UESRR7PRYC6NCXJOA, seg_pos={0 0}, upload=439960]: Long tail cancellation due to optimal threshold Dec 26 12:47:34.951971644 (501h18m34.823054709s) [noise=true , address=45.91.116.8:4074 , piece_id=YDTDZNKX5LRO3V3UQHWQB64PS7OBRMIZEBIJQ4TWONL5K2KRZVWA, seg_pos={0 0}, upload=439960]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12whTD1Kvu4T7ANNtkSvDB66uTjVgfPTK9TAzExgvwJYgZ3zSFG): protocol: expected piece hash; context canceled; EOF Dec 26 12:47:34.951991683 (501h18m34.823074718s) [noise=true , address=45.91.116.8:4074 , piece_id=YDTDZNKX5LRO3V3UQHWQB64PS7OBRMIZEBIJQ4TWONL5K2KRZVWA, seg_pos={0 0}, upload=439960]: Long tail cancellation due to optimal threshold Dec 26 12:47:34.951995770 (501h18m34.823078805s) [noise=true , address=45.91.116.6:4993 , piece_id=PKE5HHLE4GKQ34O3I2FONBGICCHP7TCMTLUFQIC4PIXOU5L5MBTQ, seg_pos={0 0}, upload=439960]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XP9gxzzBfzvzxZTatrKN9nLFvxsBuAioHxgNhSHmJYU8NAJfc): protocol: expected piece hash; context canceled; EOF Dec 26 12:47:34.952004647 (501h18m34.823087682s) [noise=true , address=45.91.116.6:4993 , piece_id=PKE5HHLE4GKQ34O3I2FONBGICCHP7TCMTLUFQIC4PIXOU5L5MBTQ, seg_pos={0 0}, upload=439960]: Long tail cancellation due to optimal threshold Dec 26 12:47:34.952136378 (501h18m34.823219413s) [noise=true , address=77.90.31.11:4562 , piece_id=4D42T5I4SG3PZMQNR6NYWA2OQYYRCKTYL3PP33SKHZYQSE3WO5NQ, seg_pos={0 0}, upload=439960]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r): protocol: expected piece hash; context canceled; EOF Dec 26 12:47:34.952143882 (501h18m34.823226917s) [noise=true , address=77.90.31.11:4562 , piece_id=4D42T5I4SG3PZMQNR6NYWA2OQYYRCKTYL3PP33SKHZYQSE3WO5NQ, seg_pos={0 0}, upload=439960]: Long tail cancellation due to optimal threshold Dec 26 12:47:34.952185611 (501h18m34.823268646s) [seg_pos={0 0}, upload=439960]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 12:47:34.952264482 (501h18m34.823347517s) [upload=439960]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:47:34.952270033 (501h18m34.823353068s) [upload=439960]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:47:35.254157018 (501h18m35.125240084s) [upload=439960]: Done uploading segments... Dec 26 12:47:35.254181404 (501h18m35.125264439s) [upload=439960]: Upload finished. err: <nil> Duration: 56.445226651s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [X..X......X.......XXX...X.....X.......X.XX......XXX..XXXXXXX] duration: 5.00129425s trace: https://jaeger.dev.storj.io/trace/364fd33ca62f4be4 FAIL (2025-12-26 12:49:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00132027s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [...X..X...X..XX.....XXXX.....XXX....XX....X..X......X.X..XX.] duration: 5.711035162s trace: https://jaeger.dev.storj.io/trace/a7946c4655f3019 FAIL (2025-12-26 12:49:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: 67B4CJKGQOCJNRJK3MKI3T6B4I6A6BFRIS4ZIU5LOATZWTZ5O3TA) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: KAGOUHT7WVAZFM3Y4RIAWDVDHF4WB3TS2IGMX2OU74B4P4B3ZXRA) context deadline exceeded ======LOGS====== Dec 26 12:49:02.447043594 (501h20m2.318126639s) [upload=439980]: Starting upload Dec 26 12:49:02.447156158 (501h20m2.318239203s) [upload=439980]: Done starting upload Dec 26 12:49:02.447204801 (501h20m2.318287846s) [upload=439980]: Uploading segments... Dec 26 12:49:02.447398079 (501h20m2.318481124s) [upload=439980]: Got next segment. Inline: false Dec 26 12:49:02.447447503 (501h20m2.318530538s) [upload=439980]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:49:02.447454657 (501h20m2.318537702s) [upload=439980]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:49:02.688830688 (501h20m2.559913744s) [seg_pos={0 0}, upload=439980]: Begin upload segment... Dec 26 12:49:02.688890753 (501h20m2.559973788s) [seg_pos={0 0}, upload=439980]: Done begin upload segment. Dec 26 12:49:02.688920770 (501h20m2.560003815s) [noise=true , address=145.79.227.45:4229 , piece_id=TRG4WAIVFL33BVBJ5PCIQ5EUPZ4LI5UL2TJLK6EN5HMVLD5J5EMQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.689093059 (501h20m2.560176094s) [noise=true , address=145.79.227.46:4602 , piece_id=CPIT2K2YAAC55GDEC5LPCNJ22SSII7MVTBUORL6X2HACU2CKOABQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.689721966 (501h20m2.560805021s) [noise=true , address=145.79.227.31:4799 , piece_id=SE75ODCMXKIUAR3JTTYRYSHF6BOWH6WMCXGRX4KWRHZPLFJDI57A, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.690107059 (501h20m2.561190104s) [noise=true , address=145.79.227.15:4876 , piece_id=2UCA4YCUUVYCOWZ4HZF3R2T7GT64Z5KZKL6N6QXVMZR26MVGDR3A, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.690704497 (501h20m2.561787542s) [noise=true , address=145.79.227.41:4087 , piece_id=L77EWW6ZI4P5KXRBD7D4WQKMZ3J5YO42LUI53FBIAOQRT4D2X5XQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.691216792 (501h20m2.562299838s) [noise=true , address=145.79.227.17:4971 , piece_id=ZEELPS5ZQTMLZ2LIPNUJ4XRV7TID4VOYPJXVUOL3DA5R3LMPU2VQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.691700504 (501h20m2.562783549s) [noise=true , address=45.91.116.14:4191 , piece_id=ANVMW7PZB54QZQCCIJTFZAMCLNHQP6M5B7SO2XDG72RNE2P5DA2Q, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.692170979 (501h20m2.563254024s) [noise=true , address=145.79.227.35:4481 , piece_id=HM2ZHXJJTEELPP76YNF3FV3YKQ4X64QJPOXDIPANOICYBJZNEW5Q, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.692728280 (501h20m2.563811325s) [noise=true , address=145.79.227.26:4654 , piece_id=QLPH4VGYQJGALFYUZOJ23OWOMOYNPIQ3MSYM6XQIA5ZFYTQ2RAWQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.694872544 (501h20m2.565955589s) [noise=true , address=145.79.227.11:4731 , piece_id=N6N2IEWXNY3B4ASST2B2EZ5375T2F6AV6JXELAYSRW4X2MCCGRTQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.697026275 (501h20m2.56810931s) [noise=true , address=45.91.116.11:4117 , piece_id=IENGJEVLVYZNAOBXBKFLAEJ3EOBYYSL7C3LGDYXYD52TGNR5R6EQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.699009491 (501h20m2.570092536s) [noise=true , address=77.90.31.16:4735 , piece_id=KAGOUHT7WVAZFM3Y4RIAWDVDHF4WB3TS2IGMX2OU74B4P4B3ZXRA, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.700818746 (501h20m2.571901791s) [noise=true , address=145.79.227.39:4014 , piece_id=VGHWBYL4OHJBVBCBPHL7DNY5O5C57OHL766P4JZEWRKPSBOEOONQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.703072998 (501h20m2.574156033s) [noise=true , address=77.90.31.10:4519 , piece_id=QCIWEIL43J22JV6ESHU2TPHBGBDPS2CMWUSVRKW4HL5XCETPQPCA, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.707037057 (501h20m2.578120102s) [noise=true , address=145.79.227.36:4518 , piece_id=URZLNXCY4VBUY4HSBWP6WKCM2RUMEGIL4UT7ZZM4QA6BJLEF7LAQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.708645239 (501h20m2.579728294s) [noise=true , address=77.90.31.14:4663 , piece_id=67B4CJKGQOCJNRJK3MKI3T6B4I6A6BFRIS4ZIU5LOATZWTZ5O3TA, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:04.416018446 (501h20m4.287101491s) [upload=439980]: Next returned nil segment Dec 26 12:49:04.416077238 (501h20m4.287160283s) [upload=439980]: Waiting for error group managing segments... Dec 26 12:49:04.703930962 (501h20m4.575014007s) [noise=true , address=145.79.227.15:4876 , piece_id=2UCA4YCUUVYCOWZ4HZF3R2T7GT64Z5KZKL6N6QXVMZR26MVGDR3A, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.704247905 (501h20m4.57533096s) [noise=true , address=145.79.227.35:4481 , piece_id=HM2ZHXJJTEELPP76YNF3FV3YKQ4X64QJPOXDIPANOICYBJZNEW5Q, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.707534473 (501h20m4.578617508s) [noise=true , address=145.79.227.26:4654 , piece_id=QLPH4VGYQJGALFYUZOJ23OWOMOYNPIQ3MSYM6XQIA5ZFYTQ2RAWQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.707569971 (501h20m4.578653006s) [noise=true , address=145.79.227.11:4731 , piece_id=N6N2IEWXNY3B4ASST2B2EZ5375T2F6AV6JXELAYSRW4X2MCCGRTQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.713069783 (501h20m4.584152828s) [noise=true , address=145.79.227.17:4971 , piece_id=ZEELPS5ZQTMLZ2LIPNUJ4XRV7TID4VOYPJXVUOL3DA5R3LMPU2VQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.715337060 (501h20m4.586420106s) [noise=true , address=145.79.227.46:4602 , piece_id=CPIT2K2YAAC55GDEC5LPCNJ22SSII7MVTBUORL6X2HACU2CKOABQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.719134893 (501h20m4.590217938s) [noise=true , address=145.79.227.45:4229 , piece_id=TRG4WAIVFL33BVBJ5PCIQ5EUPZ4LI5UL2TJLK6EN5HMVLD5J5EMQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.721251483 (501h20m4.592334518s) [noise=true , address=145.79.227.36:4518 , piece_id=URZLNXCY4VBUY4HSBWP6WKCM2RUMEGIL4UT7ZZM4QA6BJLEF7LAQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.721582844 (501h20m4.592665889s) [noise=true , address=145.79.227.41:4087 , piece_id=L77EWW6ZI4P5KXRBD7D4WQKMZ3J5YO42LUI53FBIAOQRT4D2X5XQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.728346563 (501h20m4.599429608s) [noise=true , address=145.79.227.31:4799 , piece_id=SE75ODCMXKIUAR3JTTYRYSHF6BOWH6WMCXGRX4KWRHZPLFJDI57A, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.733132587 (501h20m4.604215632s) [noise=true , address=145.79.227.39:4014 , piece_id=VGHWBYL4OHJBVBCBPHL7DNY5O5C57OHL766P4JZEWRKPSBOEOONQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.978493166 (501h20m4.849576202s) [noise=true , address=77.90.31.16:4735 , piece_id=KAGOUHT7WVAZFM3Y4RIAWDVDHF4WB3TS2IGMX2OU74B4P4B3ZXRA, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.978960205 (501h20m4.85004326s) [noise=true , address=77.90.31.14:4663 , piece_id=67B4CJKGQOCJNRJK3MKI3T6B4I6A6BFRIS4ZIU5LOATZWTZ5O3TA, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.978982949 (501h20m4.850065994s) [seg_pos={0 0}, upload=439980]: Segment reached optimal threshold of 13 pieces. Dec 26 12:49:04.979234939 (501h20m4.850317984s) [noise=true , address=45.91.116.11:4117 , piece_id=IENGJEVLVYZNAOBXBKFLAEJ3EOBYYSL7C3LGDYXYD52TGNR5R6EQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 26 12:49:04.979244006 (501h20m4.850327051s) [noise=true , address=45.91.116.11:4117 , piece_id=IENGJEVLVYZNAOBXBKFLAEJ3EOBYYSL7C3LGDYXYD52TGNR5R6EQ, seg_pos={0 0}, upload=439980]: Long tail cancellation due to optimal threshold Dec 26 12:49:04.979607709 (501h20m4.850690754s) [noise=true , address=77.90.31.10:4519 , piece_id=QCIWEIL43J22JV6ESHU2TPHBGBDPS2CMWUSVRKW4HL5XCETPQPCA, seg_pos={0 0}, upload=439980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 26 12:49:04.979615263 (501h20m4.850698298s) [noise=true , address=77.90.31.10:4519 , piece_id=QCIWEIL43J22JV6ESHU2TPHBGBDPS2CMWUSVRKW4HL5XCETPQPCA, seg_pos={0 0}, upload=439980]: Long tail cancellation due to optimal threshold Dec 26 12:49:04.979736964 (501h20m4.850820009s) [noise=true , address=45.91.116.14:4191 , piece_id=ANVMW7PZB54QZQCCIJTFZAMCLNHQP6M5B7SO2XDG72RNE2P5DA2Q, seg_pos={0 0}, upload=439980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:59208->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:59208->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59208->45.91.116.14:4191: use of closed network connection Dec 26 12:49:04.979748377 (501h20m4.850831412s) [noise=true , address=45.91.116.14:4191 , piece_id=ANVMW7PZB54QZQCCIJTFZAMCLNHQP6M5B7SO2XDG72RNE2P5DA2Q, seg_pos={0 0}, upload=439980]: Long tail cancellation due to optimal threshold Dec 26 12:49:04.979769035 (501h20m4.85085207s) [seg_pos={0 0}, upload=439980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:49:04.979846002 (501h20m4.850929047s) [upload=439980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:49:04.979851082 (501h20m4.850934117s) [upload=439980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:49:05.271932757 (501h20m5.143015792s) [upload=439980]: Done uploading segments... Dec 26 12:49:05.271960820 (501h20m5.143043855s) [upload=439980]: Upload finished. err: <nil> Duration: 5.711069677s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [....X..X...X..XX.....XXXX.....XXX....XX....X..X......X.X..XX] duration: 30.003186288s trace: https://jaeger.dev.storj.io/trace/a7946c4655f3019 FAIL (2025-12-26 12:49:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: 67B4CJKGQOCJNRJK3MKI3T6B4I6A6BFRIS4ZIU5LOATZWTZ5O3TA) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: KAGOUHT7WVAZFM3Y4RIAWDVDHF4WB3TS2IGMX2OU74B4P4B3ZXRA) context deadline exceeded ======LOGS====== Dec 26 12:49:02.447043594 (501h20m2.318126639s) [upload=439980]: Starting upload Dec 26 12:49:02.447156158 (501h20m2.318239203s) [upload=439980]: Done starting upload Dec 26 12:49:02.447204801 (501h20m2.318287846s) [upload=439980]: Uploading segments... Dec 26 12:49:02.447398079 (501h20m2.318481124s) [upload=439980]: Got next segment. Inline: false Dec 26 12:49:02.447447503 (501h20m2.318530538s) [upload=439980]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:49:02.447454657 (501h20m2.318537702s) [upload=439980]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:49:02.688830688 (501h20m2.559913744s) [seg_pos={0 0}, upload=439980]: Begin upload segment... Dec 26 12:49:02.688890753 (501h20m2.559973788s) [seg_pos={0 0}, upload=439980]: Done begin upload segment. Dec 26 12:49:02.688920770 (501h20m2.560003815s) [noise=true , address=145.79.227.45:4229 , piece_id=TRG4WAIVFL33BVBJ5PCIQ5EUPZ4LI5UL2TJLK6EN5HMVLD5J5EMQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.689093059 (501h20m2.560176094s) [noise=true , address=145.79.227.46:4602 , piece_id=CPIT2K2YAAC55GDEC5LPCNJ22SSII7MVTBUORL6X2HACU2CKOABQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.689721966 (501h20m2.560805021s) [noise=true , address=145.79.227.31:4799 , piece_id=SE75ODCMXKIUAR3JTTYRYSHF6BOWH6WMCXGRX4KWRHZPLFJDI57A, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.690107059 (501h20m2.561190104s) [noise=true , address=145.79.227.15:4876 , piece_id=2UCA4YCUUVYCOWZ4HZF3R2T7GT64Z5KZKL6N6QXVMZR26MVGDR3A, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.690704497 (501h20m2.561787542s) [noise=true , address=145.79.227.41:4087 , piece_id=L77EWW6ZI4P5KXRBD7D4WQKMZ3J5YO42LUI53FBIAOQRT4D2X5XQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.691216792 (501h20m2.562299838s) [noise=true , address=145.79.227.17:4971 , piece_id=ZEELPS5ZQTMLZ2LIPNUJ4XRV7TID4VOYPJXVUOL3DA5R3LMPU2VQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.691700504 (501h20m2.562783549s) [noise=true , address=45.91.116.14:4191 , piece_id=ANVMW7PZB54QZQCCIJTFZAMCLNHQP6M5B7SO2XDG72RNE2P5DA2Q, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.692170979 (501h20m2.563254024s) [noise=true , address=145.79.227.35:4481 , piece_id=HM2ZHXJJTEELPP76YNF3FV3YKQ4X64QJPOXDIPANOICYBJZNEW5Q, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.692728280 (501h20m2.563811325s) [noise=true , address=145.79.227.26:4654 , piece_id=QLPH4VGYQJGALFYUZOJ23OWOMOYNPIQ3MSYM6XQIA5ZFYTQ2RAWQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.694872544 (501h20m2.565955589s) [noise=true , address=145.79.227.11:4731 , piece_id=N6N2IEWXNY3B4ASST2B2EZ5375T2F6AV6JXELAYSRW4X2MCCGRTQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.697026275 (501h20m2.56810931s) [noise=true , address=45.91.116.11:4117 , piece_id=IENGJEVLVYZNAOBXBKFLAEJ3EOBYYSL7C3LGDYXYD52TGNR5R6EQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.699009491 (501h20m2.570092536s) [noise=true , address=77.90.31.16:4735 , piece_id=KAGOUHT7WVAZFM3Y4RIAWDVDHF4WB3TS2IGMX2OU74B4P4B3ZXRA, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.700818746 (501h20m2.571901791s) [noise=true , address=145.79.227.39:4014 , piece_id=VGHWBYL4OHJBVBCBPHL7DNY5O5C57OHL766P4JZEWRKPSBOEOONQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.703072998 (501h20m2.574156033s) [noise=true , address=77.90.31.10:4519 , piece_id=QCIWEIL43J22JV6ESHU2TPHBGBDPS2CMWUSVRKW4HL5XCETPQPCA, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.707037057 (501h20m2.578120102s) [noise=true , address=145.79.227.36:4518 , piece_id=URZLNXCY4VBUY4HSBWP6WKCM2RUMEGIL4UT7ZZM4QA6BJLEF7LAQ, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:02.708645239 (501h20m2.579728294s) [noise=true , address=77.90.31.14:4663 , piece_id=67B4CJKGQOCJNRJK3MKI3T6B4I6A6BFRIS4ZIU5LOATZWTZ5O3TA, seg_pos={0 0}, upload=439980]: Uploading piece... Dec 26 12:49:04.416018446 (501h20m4.287101491s) [upload=439980]: Next returned nil segment Dec 26 12:49:04.416077238 (501h20m4.287160283s) [upload=439980]: Waiting for error group managing segments... Dec 26 12:49:04.703930962 (501h20m4.575014007s) [noise=true , address=145.79.227.15:4876 , piece_id=2UCA4YCUUVYCOWZ4HZF3R2T7GT64Z5KZKL6N6QXVMZR26MVGDR3A, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.704247905 (501h20m4.57533096s) [noise=true , address=145.79.227.35:4481 , piece_id=HM2ZHXJJTEELPP76YNF3FV3YKQ4X64QJPOXDIPANOICYBJZNEW5Q, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.707534473 (501h20m4.578617508s) [noise=true , address=145.79.227.26:4654 , piece_id=QLPH4VGYQJGALFYUZOJ23OWOMOYNPIQ3MSYM6XQIA5ZFYTQ2RAWQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.707569971 (501h20m4.578653006s) [noise=true , address=145.79.227.11:4731 , piece_id=N6N2IEWXNY3B4ASST2B2EZ5375T2F6AV6JXELAYSRW4X2MCCGRTQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.713069783 (501h20m4.584152828s) [noise=true , address=145.79.227.17:4971 , piece_id=ZEELPS5ZQTMLZ2LIPNUJ4XRV7TID4VOYPJXVUOL3DA5R3LMPU2VQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.715337060 (501h20m4.586420106s) [noise=true , address=145.79.227.46:4602 , piece_id=CPIT2K2YAAC55GDEC5LPCNJ22SSII7MVTBUORL6X2HACU2CKOABQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.719134893 (501h20m4.590217938s) [noise=true , address=145.79.227.45:4229 , piece_id=TRG4WAIVFL33BVBJ5PCIQ5EUPZ4LI5UL2TJLK6EN5HMVLD5J5EMQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.721251483 (501h20m4.592334518s) [noise=true , address=145.79.227.36:4518 , piece_id=URZLNXCY4VBUY4HSBWP6WKCM2RUMEGIL4UT7ZZM4QA6BJLEF7LAQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.721582844 (501h20m4.592665889s) [noise=true , address=145.79.227.41:4087 , piece_id=L77EWW6ZI4P5KXRBD7D4WQKMZ3J5YO42LUI53FBIAOQRT4D2X5XQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.728346563 (501h20m4.599429608s) [noise=true , address=145.79.227.31:4799 , piece_id=SE75ODCMXKIUAR3JTTYRYSHF6BOWH6WMCXGRX4KWRHZPLFJDI57A, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.733132587 (501h20m4.604215632s) [noise=true , address=145.79.227.39:4014 , piece_id=VGHWBYL4OHJBVBCBPHL7DNY5O5C57OHL766P4JZEWRKPSBOEOONQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.978493166 (501h20m4.849576202s) [noise=true , address=77.90.31.16:4735 , piece_id=KAGOUHT7WVAZFM3Y4RIAWDVDHF4WB3TS2IGMX2OU74B4P4B3ZXRA, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.978960205 (501h20m4.85004326s) [noise=true , address=77.90.31.14:4663 , piece_id=67B4CJKGQOCJNRJK3MKI3T6B4I6A6BFRIS4ZIU5LOATZWTZ5O3TA, seg_pos={0 0}, upload=439980]: Done uploading piece. err: <nil> Dec 26 12:49:04.978982949 (501h20m4.850065994s) [seg_pos={0 0}, upload=439980]: Segment reached optimal threshold of 13 pieces. Dec 26 12:49:04.979234939 (501h20m4.850317984s) [noise=true , address=45.91.116.11:4117 , piece_id=IENGJEVLVYZNAOBXBKFLAEJ3EOBYYSL7C3LGDYXYD52TGNR5R6EQ, seg_pos={0 0}, upload=439980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 26 12:49:04.979244006 (501h20m4.850327051s) [noise=true , address=45.91.116.11:4117 , piece_id=IENGJEVLVYZNAOBXBKFLAEJ3EOBYYSL7C3LGDYXYD52TGNR5R6EQ, seg_pos={0 0}, upload=439980]: Long tail cancellation due to optimal threshold Dec 26 12:49:04.979607709 (501h20m4.850690754s) [noise=true , address=77.90.31.10:4519 , piece_id=QCIWEIL43J22JV6ESHU2TPHBGBDPS2CMWUSVRKW4HL5XCETPQPCA, seg_pos={0 0}, upload=439980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 26 12:49:04.979615263 (501h20m4.850698298s) [noise=true , address=77.90.31.10:4519 , piece_id=QCIWEIL43J22JV6ESHU2TPHBGBDPS2CMWUSVRKW4HL5XCETPQPCA, seg_pos={0 0}, upload=439980]: Long tail cancellation due to optimal threshold Dec 26 12:49:04.979736964 (501h20m4.850820009s) [noise=true , address=45.91.116.14:4191 , piece_id=ANVMW7PZB54QZQCCIJTFZAMCLNHQP6M5B7SO2XDG72RNE2P5DA2Q, seg_pos={0 0}, upload=439980]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:59208->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:59208->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59208->45.91.116.14:4191: use of closed network connection Dec 26 12:49:04.979748377 (501h20m4.850831412s) [noise=true , address=45.91.116.14:4191 , piece_id=ANVMW7PZB54QZQCCIJTFZAMCLNHQP6M5B7SO2XDG72RNE2P5DA2Q, seg_pos={0 0}, upload=439980]: Long tail cancellation due to optimal threshold Dec 26 12:49:04.979769035 (501h20m4.85085207s) [seg_pos={0 0}, upload=439980]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:49:04.979846002 (501h20m4.850929047s) [upload=439980]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:49:04.979851082 (501h20m4.850934117s) [upload=439980]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:49:05.271932757 (501h20m5.143015792s) [upload=439980]: Done uploading segments... Dec 26 12:49:05.271960820 (501h20m5.143043855s) [upload=439980]: Upload finished. err: <nil> Duration: 30.003202148s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............................X....................X.XXX...X.] duration: 58.257657938s trace: https://jaeger.dev.storj.io/trace/55808baa5e3da7b9 FAIL (2025-12-26 12:49:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gjesgEXSFr1aGyFgASrVRJgEHTne1KichuGqXwMjN1hHBt4kz, Piece ID: QOXGFQE7CP3745MSKKMAYIDMD4CEGI642K2QGF2Q5TYQHYJ5H7YA) context deadline exceeded; piecestore: (Node ID: 12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, Piece ID: PPFXUZ7QDC33NCS6PMM4Z4SWBGFYPGJPBVQYE3FLPONDFHMXLXSQ) context deadline exceeded; piecestore: (Node ID: 1AvSrhj1nHWLBJrw1SKacnjPTi9g4AzjqKvVFoxtzAdJFYAfQt, Piece ID: FQ5HW32FOAV5VHVLXIFPWWG67DVCKE5OSIDSWAEDJ7FH3USONACA) context deadline exceeded ======LOGS====== 1.996987257 (501h18m31.868070292s) [noise=true , address=145.79.227.29:4954 , piece_id=6BH4PB2YY6UMBZHSU7B4MS64PSFHC5SRFCNSYUWHLHKQTUKR7EDQ, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:31.997520121 (501h18m31.868603166s) [noise=true , address=77.90.31.19:4849 , piece_id=BB52EEC5VJKTJ7H722N7SXUMWBWAUAVFMRHMPORK3KC6QY6LMHZQ, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:31.998001308 (501h18m31.869084353s) [noise=true , address=145.79.227.30:4782 , piece_id=KAEZ3CPVWDFBFEEAKZXLFCGJ7XAYYXRQV5TMXZUS2AD6CBX3HPTA, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:31.998213993 (501h18m31.869297028s) [noise=true , address=145.79.227.44:4226 , piece_id=FQY74SGDMFVPM54JVMLG2DATBKKX4UI2JLO5C5YIUFONTAMPTJQQ, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:31.998539562 (501h18m31.869622607s) [noise=true , address=77.90.31.11:4562 , piece_id=4D42T5I4SG3PZMQNR6NYWA2OQYYRCKTYL3PP33SKHZYQSE3WO5NQ, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:31.998740024 (501h18m31.869823059s) [noise=true , address=145.79.227.43:4171 , piece_id=MDGS63GO7OPP6Y2YYCHJOZSZK25X3YTJHXDEOG4RZAF5LZ4IABZA, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:31.998960043 (501h18m31.870043078s) [noise=true , address=145.79.227.40:4069 , piece_id=ISS3ZLARB55VNPWUMFL3WXHGC3NZMKSFWPLVTPWEVZJ7EXUPF5WQ, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:31.999138473 (501h18m31.870221508s) [noise=true , address=45.91.116.6:4993 , piece_id=PKE5HHLE4GKQ34O3I2FONBGICCHP7TCMTLUFQIC4PIXOU5L5MBTQ, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:31.999374642 (501h18m31.870457677s) [noise=true , address=45.91.116.4:4962 , piece_id=P5SBO3YE6QCFXRVAL62G3HC5XJDNJBPA3PMQEYLNB2U7M46IXKCA, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:31.999529438 (501h18m31.870612483s) [noise=true , address=145.79.227.21:4486 , piece_id=V7DWPLGDLWDYE4TJLGIFMTOLDCXEDBTPUYSWYXPFXOLG3KZDKDQA, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:31.999937926 (501h18m31.871020961s) [noise=true , address=145.79.227.34:4939 , piece_id=FAEDI2PHQ543A5OOP6K73ZDWCM3TPTQQBJPFEYOAGW26J7GX5S6Q, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:32.000931908 (501h18m31.872014943s) [noise=true , address=77.90.31.15:4708 , piece_id=PPFXUZ7QDC33NCS6PMM4Z4SWBGFYPGJPBVQYE3FLPONDFHMXLXSQ, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:32.001377096 (501h18m31.872460141s) [noise=true , address=77.90.31.16:4757 , piece_id=QOXGFQE7CP3745MSKKMAYIDMD4CEGI642K2QGF2Q5TYQHYJ5H7YA, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:32.001947612 (501h18m31.873030647s) [noise=true , address=145.79.227.28:4750 , piece_id=SYOPLUXNCR7KCMR7FX6PMICD4SDW7VLAKNWRWVC3455ESX4P4IKQ, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:32.002400785 (501h18m31.87348383s) [noise=true , address=45.91.116.8:4074 , piece_id=YDTDZNKX5LRO3V3UQHWQB64PS7OBRMIZEBIJQ4TWONL5K2KRZVWA, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:32.002666842 (501h18m31.873749877s) [noise=true , address=45.91.116.11:4146 , piece_id=X7UK2OJXVANXFXYCWRZ6M3UIYULU4DFJ4I2UESRR7PRYC6NCXJOA, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:32.003349923 (501h18m31.874432968s) [noise=true , address=77.90.31.14:4671 , piece_id=O7VFBBQLQ5FBP4UPQFELDDKB5JR4GUEFY6I62BVS4D7VCXRT7R5A, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:33.469721230 (501h18m33.340804265s) [upload=439960]: Next returned nil segment Dec 26 12:47:33.469757930 (501h18m33.340840975s) [upload=439960]: Waiting for error group managing segments... Dec 26 12:47:33.864743802 (501h18m33.735826837s) [noise=true , address=145.79.227.29:4954 , piece_id=6BH4PB2YY6UMBZHSU7B4MS64PSFHC5SRFCNSYUWHLHKQTUKR7EDQ, seg_pos={0 0}, upload=439960]: Done uploading piece. err: <nil> Dec 26 12:47:33.878583104 (501h18m33.749666149s) [noise=true , address=145.79.227.21:4486 , piece_id=V7DWPLGDLWDYE4TJLGIFMTOLDCXEDBTPUYSWYXPFXOLG3KZDKDQA, seg_pos={0 0}, upload=439960]: Done uploading piece. err: <nil> Dec 26 12:47:33.885532327 (501h18m33.756615362s) [noise=true , address=145.79.227.28:4750 , piece_id=SYOPLUXNCR7KCMR7FX6PMICD4SDW7VLAKNWRWVC3455ESX4P4IKQ, seg_pos={0 0}, upload=439960]: Done uploading piece. err: <nil> Dec 26 12:47:33.889919020 (501h18m33.761002065s) [noise=true , address=145.79.227.43:4171 , piece_id=MDGS63GO7OPP6Y2YYCHJOZSZK25X3YTJHXDEOG4RZAF5LZ4IABZA, seg_pos={0 0}, upload=439960]: Done uploading piece. err: <nil> Dec 26 12:47:34.047203828 (501h18m33.918286873s) [noise=true , address=77.90.31.15:4708 , piece_id=PPFXUZ7QDC33NCS6PMM4Z4SWBGFYPGJPBVQYE3FLPONDFHMXLXSQ, seg_pos={0 0}, upload=439960]: Done uploading piece. err: <nil> Dec 26 12:47:34.070368728 (501h18m33.941451763s) [noise=true , address=45.91.116.4:4962 , piece_id=P5SBO3YE6QCFXRVAL62G3HC5XJDNJBPA3PMQEYLNB2U7M46IXKCA, seg_pos={0 0}, upload=439960]: Done uploading piece. err: <nil> Dec 26 12:47:34.091662365 (501h18m33.9627454s) [noise=true , address=77.90.31.16:4757 , piece_id=QOXGFQE7CP3745MSKKMAYIDMD4CEGI642K2QGF2Q5TYQHYJ5H7YA, seg_pos={0 0}, upload=439960]: Done uploading piece. err: <nil> Dec 26 12:47:34.143840831 (501h18m34.014923867s) [noise=true , address=77.90.31.19:4849 , piece_id=BB52EEC5VJKTJ7H722N7SXUMWBWAUAVFMRHMPORK3KC6QY6LMHZQ, seg_pos={0 0}, upload=439960]: Done uploading piece. err: <nil> Dec 26 12:47:34.601885600 (501h18m34.472968655s) [noise=true , address=77.90.31.14:4671 , piece_id=O7VFBBQLQ5FBP4UPQFELDDKB5JR4GUEFY6I62BVS4D7VCXRT7R5A, seg_pos={0 0}, upload=439960]: Done uploading piece. err: <nil> Dec 26 12:47:34.776497801 (501h18m34.647580836s) [noise=true , address=145.79.227.40:4069 , piece_id=ISS3ZLARB55VNPWUMFL3WXHGC3NZMKSFWPLVTPWEVZJ7EXUPF5WQ, seg_pos={0 0}, upload=439960]: Done uploading piece. err: <nil> Dec 26 12:47:34.777613185 (501h18m34.64869623s) [noise=true , address=145.79.227.44:4226 , piece_id=FQY74SGDMFVPM54JVMLG2DATBKKX4UI2JLO5C5YIUFONTAMPTJQQ, seg_pos={0 0}, upload=439960]: Done uploading piece. err: <nil> Dec 26 12:47:34.792547793 (501h18m34.663630828s) [noise=true , address=145.79.227.30:4782 , piece_id=KAEZ3CPVWDFBFEEAKZXLFCGJ7XAYYXRQV5TMXZUS2AD6CBX3HPTA, seg_pos={0 0}, upload=439960]: Done uploading piece. err: <nil> Dec 26 12:47:34.830745548 (501h18m34.701828583s) [noise=true , address=77.90.31.10:4531 , piece_id=FQ5HW32FOAV5VHVLXIFPWWG67DVCKE5OSIDSWAEDJ7FH3USONACA, seg_pos={0 0}, upload=439960]: Done uploading piece. err: <nil> Dec 26 12:47:34.951612160 (501h18m34.822695205s) [noise=true , address=145.79.227.34:4939 , piece_id=FAEDI2PHQ543A5OOP6K73ZDWCM3TPTQQBJPFEYOAGW26J7GX5S6Q, seg_pos={0 0}, upload=439960]: Done uploading piece. err: <nil> Dec 26 12:47:34.951634812 (501h18m34.822717857s) [seg_pos={0 0}, upload=439960]: Segment reached optimal threshold of 14 pieces. Dec 26 12:47:34.951856816 (501h18m34.822939851s) [noise=true , address=45.91.116.11:4146 , piece_id=X7UK2OJXVANXFXYCWRZ6M3UIYULU4DFJ4I2UESRR7PRYC6NCXJOA, seg_pos={0 0}, upload=439960]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp): protocol: expected piece hash; context canceled; EOF Dec 26 12:47:34.951864901 (501h18m34.822947946s) [noise=true , address=45.91.116.11:4146 , piece_id=X7UK2OJXVANXFXYCWRZ6M3UIYULU4DFJ4I2UESRR7PRYC6NCXJOA, seg_pos={0 0}, upload=439960]: Long tail cancellation due to optimal threshold Dec 26 12:47:34.951971644 (501h18m34.823054709s) [noise=true , address=45.91.116.8:4074 , piece_id=YDTDZNKX5LRO3V3UQHWQB64PS7OBRMIZEBIJQ4TWONL5K2KRZVWA, seg_pos={0 0}, upload=439960]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12whTD1Kvu4T7ANNtkSvDB66uTjVgfPTK9TAzExgvwJYgZ3zSFG): protocol: expected piece hash; context canceled; EOF Dec 26 12:47:34.951991683 (501h18m34.823074718s) [noise=true , address=45.91.116.8:4074 , piece_id=YDTDZNKX5LRO3V3UQHWQB64PS7OBRMIZEBIJQ4TWONL5K2KRZVWA, seg_pos={0 0}, upload=439960]: Long tail cancellation due to optimal threshold Dec 26 12:47:34.951995770 (501h18m34.823078805s) [noise=true , address=45.91.116.6:4993 , piece_id=PKE5HHLE4GKQ34O3I2FONBGICCHP7TCMTLUFQIC4PIXOU5L5MBTQ, seg_pos={0 0}, upload=439960]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XP9gxzzBfzvzxZTatrKN9nLFvxsBuAioHxgNhSHmJYU8NAJfc): protocol: expected piece hash; context canceled; EOF Dec 26 12:47:34.952004647 (501h18m34.823087682s) [noise=true , address=45.91.116.6:4993 , piece_id=PKE5HHLE4GKQ34O3I2FONBGICCHP7TCMTLUFQIC4PIXOU5L5MBTQ, seg_pos={0 0}, upload=439960]: Long tail cancellation due to optimal threshold Dec 26 12:47:34.952136378 (501h18m34.823219413s) [noise=true , address=77.90.31.11:4562 , piece_id=4D42T5I4SG3PZMQNR6NYWA2OQYYRCKTYL3PP33SKHZYQSE3WO5NQ, seg_pos={0 0}, upload=439960]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r): protocol: expected piece hash; context canceled; EOF Dec 26 12:47:34.952143882 (501h18m34.823226917s) [noise=true , address=77.90.31.11:4562 , piece_id=4D42T5I4SG3PZMQNR6NYWA2OQYYRCKTYL3PP33SKHZYQSE3WO5NQ, seg_pos={0 0}, upload=439960]: Long tail cancellation due to optimal threshold Dec 26 12:47:34.952185611 (501h18m34.823268646s) [seg_pos={0 0}, upload=439960]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 12:47:34.952264482 (501h18m34.823347517s) [upload=439960]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:47:34.952270033 (501h18m34.823353068s) [upload=439960]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:47:35.254157018 (501h18m35.125240084s) [upload=439960]: Done uploading segments... Dec 26 12:47:35.254181404 (501h18m35.125264439s) [upload=439960]: Upload finished. err: <nil> Duration: 58.257673357s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X....X..X...X..XX.....XXXX.....XXX....XX....X..X......X.X..X] duration: 30.003097679s trace: https://jaeger.dev.storj.io/trace/2af521b910b5f8ce FAIL (2025-12-26 12:49:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: E2X3UXTRFODBX4X2AMG3H7N3DO3F7SK3E6PXW7CZ2GHSZBYCSOYQ) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: M3NNDED4LDGOKSQVSHYOD6HDCPOQVAEXIL4A4SE6QG64GX7H6IHA) context deadline exceeded ======LOGS====== Dec 26 12:48:32.222081275 (501h19m32.09316432s) [upload=439973]: Starting upload Dec 26 12:48:32.222154194 (501h19m32.093237239s) [upload=439973]: Done starting upload Dec 26 12:48:32.222355316 (501h19m32.093438361s) [upload=439973]: Uploading segments... Dec 26 12:48:32.222639178 (501h19m32.093722223s) [upload=439973]: Got next segment. Inline: false Dec 26 12:48:32.223405767 (501h19m32.094488802s) [upload=439973]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:48:32.223415335 (501h19m32.09449838s) [upload=439973]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:48:32.467250370 (501h19m32.338333415s) [seg_pos={0 0}, upload=439973]: Begin upload segment... Dec 26 12:48:32.467317868 (501h19m32.338400903s) [seg_pos={0 0}, upload=439973]: Done begin upload segment. Dec 26 12:48:32.467365760 (501h19m32.338448795s) [noise=true , address=145.79.227.52:4445 , piece_id=IA5LB2C7GOQACFZG5KKA2NWIKDDF3JVQF62MYEB6QUX22QVXAQQQ, seg_pos={0 0}, upload=439973]: Uploading piece... Dec 26 12:48:32.467422457 (501h19m32.338505512s) [noise=true , address=45.91.116.11:4117 , piece_id=4CKDPZ2KNYVMAPIM5F25OLOLAWX3DL43L3RIFYQKZL26P4UV775A, seg_pos={0 0}, upload=439973]: Uploading piece... Dec 26 12:48:32.468360934 (501h19m32.33944398s) [noise=true , address=45.91.116.9:4262 , piece_id=XM2LJXRAOLCGXVA2ACUKSOQ2OVUEBTKU7KJNYIZ7AKX646NX2R5Q, seg_pos={0 0}, upload=439973]: Uploading piece... Dec 26 12:48:32.468818636 (501h19m32.339901681s) [noise=true , address=45.91.116.8:4046 , piece_id=E5SPDYU4ARXWRXV3OZVPGUQE4FMI43TH7SANKLHT75E7TK2M2WKA, seg_pos={0 0}, upload=439973]: Uploading piece... Dec 26 12:48:32.469051880 (501h19m32.340134915s) [noise=true , address=145.79.227.48:4303 , piece_id=YDHUSAGXPMQ4WYCM77KL5LP2UGJNKLRRGNVCG4LVHMEQL7KBD2OA, seg_pos={0 0}, upload=439973]: Uploading piece... Dec 26 12:48:32.469181277 (501h19m32.340264312s) [noise=true , address=45.91.116.14:4191 , piece_id=E2X3UXTRFODBX4X2AMG3H7N3DO3F7SK3E6PXW7CZ2GHSZBYCSOYQ, seg_pos={0 0}, upload=439973]: Uploading piece... Dec 26 12:48:32.469558696 (501h19m32.340641731s) [noise=true , address=45.91.116.7:4012 , piece_id=NMFIERRN4C3HGH2S6STOXUN4GAVNI4O2QOTV4TGG7647TODZKTCQ, seg_pos={0 0}, upload=439973]: Uploading piece... Dec 26 12:48:32.470037478 (501h19m32.341120523s) [noise=true , address=145.79.227.49:4337 , piece_id=LXG27GQG2PSFMTRYVFLHZ2Z7CZXTW73MJQPIIVDQHZRGQZWR7CDQ, seg_pos={0 0}, upload=439973]: Uploading piece... Dec 26 12:48:32.470143049 (501h19m32.341226084s) [noise=true , address=45.91.116.5:4300 , piece_id=M3NNDED4LDGOKSQVSHYOD6HDCPOQVAEXIL4A4SE6QG64GX7H6IHA, seg_pos={0 0}, upload=439973]: Uploading piece... Dec 26 12:48:32.470565583 (501h19m32.341648628s) [noise=true , address=145.79.227.38:4977 , piece_id=5GGT3YDI7P2Q76KMSVGFGJFV7FUCWHUUKFMMC5TC62PUTSYRQHVQ, seg_pos={0 0}, upload=439973]: Uploading piece... Dec 26 12:48:32.470974201 (501h19m32.342057236s) [noise=true , address=145.79.227.44:4194 , piece_id=S7UQS3X2YOK24GCQ7JD2AYBBRYYBCH75C22LRCHTEMFGWLGAIA3A, seg_pos={0 0}, upload=439973]: Uploading piece... Dec 26 12:48:32.471160015 (501h19m32.34224305s) [noise=true , address=145.79.227.23:4546 , piece_id=3JSBGJVRX4YM5JZD2RNXLEWUHE2CXV4CQZI5VDLA4ISUCELJPZ4Q, seg_pos={0 0}, upload=439973]: Uploading piece... Dec 26 12:48:32.471216682 (501h19m32.342299717s) [noise=true , address=145.79.227.46:4603 , piece_id=AWZDOWI4T6UJBCEYQ4JEDH6S2YB2AGPLYVLADGX3RHBZOP4O7EUQ, seg_pos={0 0}, upload=439973]: Uploading piece... Dec 26 12:48:32.471251799 (501h19m32.342334834s) [noise=true , address=145.79.227.2:4114 , piece_id=AT35SSZOE7FJIRI52LRVFEG74PHOZUM4RSERPJABQHZEKJ2N4JEA, seg_pos={0 0}, upload=439973]: Uploading piece... Dec 26 12:48:32.472818863 (501h19m32.343901908s) [noise=true , address=45.91.116.10:4082 , piece_id=TANTRN23HDFJIF3ZKQG4PVTGSSUASYSL3WC6BHNFJMRPVC6TNPHQ, seg_pos={0 0}, upload=439973]: Uploading piece... Dec 26 12:48:32.475775844 (501h19m32.346858879s) [noise=true , address=145.79.227.15:4876 , piece_id=AMUHOUOIVM5XTMGTTXHHEOKZDBSH34FA6IRQ56VK2E7BNDO5UGTQ, seg_pos={0 0}, upload=439973]: Uploading piece... Dec 26 12:48:34.188721722 (501h19m34.059804757s) [upload=439973]: Next returned nil segment Dec 26 12:48:34.188768051 (501h19m34.059851096s) [upload=439973]: Waiting for error group managing segments... Dec 26 12:48:34.475307121 (501h19m34.346390166s) [noise=true , address=145.79.227.48:4303 , piece_id=YDHUSAGXPMQ4WYCM77KL5LP2UGJNKLRRGNVCG4LVHMEQL7KBD2OA, seg_pos={0 0}, upload=439973]: Done uploading piece. err: <nil> Dec 26 12:48:34.485287857 (501h19m34.356370892s) [noise=true , address=145.79.227.49:4337 , piece_id=LXG27GQG2PSFMTRYVFLHZ2Z7CZXTW73MJQPIIVDQHZRGQZWR7CDQ, seg_pos={0 0}, upload=439973]: Done uploading piece. err: <nil> Dec 26 12:48:34.487527050 (501h19m34.358610095s) [noise=true , address=145.79.227.52:4445 , piece_id=IA5LB2C7GOQACFZG5KKA2NWIKDDF3JVQF62MYEB6QUX22QVXAQQQ, seg_pos={0 0}, upload=439973]: Done uploading piece. err: <nil> Dec 26 12:48:34.487761317 (501h19m34.358844352s) [noise=true , address=145.79.227.15:4876 , piece_id=AMUHOUOIVM5XTMGTTXHHEOKZDBSH34FA6IRQ56VK2E7BNDO5UGTQ, seg_pos={0 0}, upload=439973]: Done uploading piece. err: <nil> Dec 26 12:48:34.499749754 (501h19m34.370832799s) [noise=true , address=145.79.227.38:4977 , piece_id=5GGT3YDI7P2Q76KMSVGFGJFV7FUCWHUUKFMMC5TC62PUTSYRQHVQ, seg_pos={0 0}, upload=439973]: Done uploading piece. err: <nil> Dec 26 12:48:34.503640112 (501h19m34.374723177s) [noise=true , address=145.79.227.46:4603 , piece_id=AWZDOWI4T6UJBCEYQ4JEDH6S2YB2AGPLYVLADGX3RHBZOP4O7EUQ, seg_pos={0 0}, upload=439973]: Done uploading piece. err: <nil> Dec 26 12:48:34.523251095 (501h19m34.39433414s) [noise=true , address=145.79.227.23:4546 , piece_id=3JSBGJVRX4YM5JZD2RNXLEWUHE2CXV4CQZI5VDLA4ISUCELJPZ4Q, seg_pos={0 0}, upload=439973]: Done uploading piece. err: <nil> Dec 26 12:48:34.526126550 (501h19m34.397209596s) [noise=true , address=145.79.227.2:4114 , piece_id=AT35SSZOE7FJIRI52LRVFEG74PHOZUM4RSERPJABQHZEKJ2N4JEA, seg_pos={0 0}, upload=439973]: Done uploading piece. err: <nil> Dec 26 12:48:34.822282052 (501h19m34.693365097s) [noise=true , address=45.91.116.11:4117 , piece_id=4CKDPZ2KNYVMAPIM5F25OLOLAWX3DL43L3RIFYQKZL26P4UV775A, seg_pos={0 0}, upload=439973]: Done uploading piece. err: <nil> Dec 26 12:48:34.832856197 (501h19m34.703939232s) [noise=true , address=45.91.116.8:4046 , piece_id=E5SPDYU4ARXWRXV3OZVPGUQE4FMI43TH7SANKLHT75E7TK2M2WKA, seg_pos={0 0}, upload=439973]: Done uploading piece. err: <nil> Dec 26 12:48:34.841507782 (501h19m34.712590827s) [noise=true , address=45.91.116.14:4191 , piece_id=E2X3UXTRFODBX4X2AMG3H7N3DO3F7SK3E6PXW7CZ2GHSZBYCSOYQ, seg_pos={0 0}, upload=439973]: Done uploading piece. err: <nil> Dec 26 12:48:34.846857728 (501h19m34.717940763s) [noise=true , address=45.91.116.10:4082 , piece_id=TANTRN23HDFJIF3ZKQG4PVTGSSUASYSL3WC6BHNFJMRPVC6TNPHQ, seg_pos={0 0}, upload=439973]: Done uploading piece. err: <nil> Dec 26 12:48:34.963844562 (501h19m34.834927597s) [noise=true , address=45.91.116.5:4300 , piece_id=M3NNDED4LDGOKSQVSHYOD6HDCPOQVAEXIL4A4SE6QG64GX7H6IHA, seg_pos={0 0}, upload=439973]: Done uploading piece. err: <nil> Dec 26 12:48:34.963863048 (501h19m34.834946083s) [seg_pos={0 0}, upload=439973]: Segment reached optimal threshold of 13 pieces. Dec 26 12:48:34.964139314 (501h19m34.835222349s) [noise=true , address=45.91.116.7:4012 , piece_id=NMFIERRN4C3HGH2S6STOXUN4GAVNI4O2QOTV4TGG7647TODZKTCQ, seg_pos={0 0}, upload=439973]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: read tcp 188.166.225.170:37340->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 12:48:34.964147770 (501h19m34.835230805s) [noise=true , address=45.91.116.7:4012 , piece_id=NMFIERRN4C3HGH2S6STOXUN4GAVNI4O2QOTV4TGG7647TODZKTCQ, seg_pos={0 0}, upload=439973]: Long tail cancellation due to optimal threshold Dec 26 12:48:34.964182165 (501h19m34.83526521s) [noise=true , address=45.91.116.9:4262 , piece_id=XM2LJXRAOLCGXVA2ACUKSOQ2OVUEBTKU7KJNYIZ7AKX646NX2R5Q, seg_pos={0 0}, upload=439973]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 26 12:48:34.964190060 (501h19m34.835273095s) [noise=true , address=45.91.116.9:4262 , piece_id=XM2LJXRAOLCGXVA2ACUKSOQ2OVUEBTKU7KJNYIZ7AKX646NX2R5Q, seg_pos={0 0}, upload=439973]: Long tail cancellation due to optimal threshold Dec 26 12:48:34.964360875 (501h19m34.83544392s) [noise=true , address=145.79.227.44:4194 , piece_id=S7UQS3X2YOK24GCQ7JD2AYBBRYYBCH75C22LRCHTEMFGWLGAIA3A, seg_pos={0 0}, upload=439973]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 26 12:48:34.964368179 (501h19m34.835451214s) [noise=true , address=145.79.227.44:4194 , piece_id=S7UQS3X2YOK24GCQ7JD2AYBBRYYBCH75C22LRCHTEMFGWLGAIA3A, seg_pos={0 0}, upload=439973]: Long tail cancellation due to optimal threshold Dec 26 12:48:34.964398527 (501h19m34.835481572s) [seg_pos={0 0}, upload=439973]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:48:34.964487557 (501h19m34.835570592s) [upload=439973]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:48:34.964492516 (501h19m34.835575561s) [upload=439973]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:48:35.252279794 (501h19m35.123362829s) [upload=439973]: Done uploading segments... Dec 26 12:48:35.252299761 (501h19m35.123382796s) [upload=439973]: Upload finished. err: <nil> Duration: 30.003120863s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [.X..X......X.......XXX...X.....X.......X.XX......XXX..XXXXXX] duration: 5.001122873s trace: https://jaeger.dev.storj.io/trace/9640a72bb28ecc FAIL (2025-12-26 12:48:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001148503s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............................X....................X.XXX...X] duration: 1m0.002934167s trace: https://jaeger.dev.storj.io/trace/55808baa5e3da7b9 FAIL (2025-12-26 12:48:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gjesgEXSFr1aGyFgASrVRJgEHTne1KichuGqXwMjN1hHBt4kz, Piece ID: QOXGFQE7CP3745MSKKMAYIDMD4CEGI642K2QGF2Q5TYQHYJ5H7YA) context deadline exceeded; piecestore: (Node ID: 12cGT35JePb3CDVwWxdHPJDPBdeLuLvMSm8Xdmx2KRMavTTbY6u, Piece ID: PPFXUZ7QDC33NCS6PMM4Z4SWBGFYPGJPBVQYE3FLPONDFHMXLXSQ) context deadline exceeded; piecestore: (Node ID: 1AvSrhj1nHWLBJrw1SKacnjPTi9g4AzjqKvVFoxtzAdJFYAfQt, Piece ID: FQ5HW32FOAV5VHVLXIFPWWG67DVCKE5OSIDSWAEDJ7FH3USONACA) context deadline exceeded ======LOGS====== 1.996987257 (501h18m31.868070292s) [noise=true , address=145.79.227.29:4954 , piece_id=6BH4PB2YY6UMBZHSU7B4MS64PSFHC5SRFCNSYUWHLHKQTUKR7EDQ, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:31.997520121 (501h18m31.868603166s) [noise=true , address=77.90.31.19:4849 , piece_id=BB52EEC5VJKTJ7H722N7SXUMWBWAUAVFMRHMPORK3KC6QY6LMHZQ, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:31.998001308 (501h18m31.869084353s) [noise=true , address=145.79.227.30:4782 , piece_id=KAEZ3CPVWDFBFEEAKZXLFCGJ7XAYYXRQV5TMXZUS2AD6CBX3HPTA, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:31.998213993 (501h18m31.869297028s) [noise=true , address=145.79.227.44:4226 , piece_id=FQY74SGDMFVPM54JVMLG2DATBKKX4UI2JLO5C5YIUFONTAMPTJQQ, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:31.998539562 (501h18m31.869622607s) [noise=true , address=77.90.31.11:4562 , piece_id=4D42T5I4SG3PZMQNR6NYWA2OQYYRCKTYL3PP33SKHZYQSE3WO5NQ, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:31.998740024 (501h18m31.869823059s) [noise=true , address=145.79.227.43:4171 , piece_id=MDGS63GO7OPP6Y2YYCHJOZSZK25X3YTJHXDEOG4RZAF5LZ4IABZA, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:31.998960043 (501h18m31.870043078s) [noise=true , address=145.79.227.40:4069 , piece_id=ISS3ZLARB55VNPWUMFL3WXHGC3NZMKSFWPLVTPWEVZJ7EXUPF5WQ, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:31.999138473 (501h18m31.870221508s) [noise=true , address=45.91.116.6:4993 , piece_id=PKE5HHLE4GKQ34O3I2FONBGICCHP7TCMTLUFQIC4PIXOU5L5MBTQ, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:31.999374642 (501h18m31.870457677s) [noise=true , address=45.91.116.4:4962 , piece_id=P5SBO3YE6QCFXRVAL62G3HC5XJDNJBPA3PMQEYLNB2U7M46IXKCA, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:31.999529438 (501h18m31.870612483s) [noise=true , address=145.79.227.21:4486 , piece_id=V7DWPLGDLWDYE4TJLGIFMTOLDCXEDBTPUYSWYXPFXOLG3KZDKDQA, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:31.999937926 (501h18m31.871020961s) [noise=true , address=145.79.227.34:4939 , piece_id=FAEDI2PHQ543A5OOP6K73ZDWCM3TPTQQBJPFEYOAGW26J7GX5S6Q, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:32.000931908 (501h18m31.872014943s) [noise=true , address=77.90.31.15:4708 , piece_id=PPFXUZ7QDC33NCS6PMM4Z4SWBGFYPGJPBVQYE3FLPONDFHMXLXSQ, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:32.001377096 (501h18m31.872460141s) [noise=true , address=77.90.31.16:4757 , piece_id=QOXGFQE7CP3745MSKKMAYIDMD4CEGI642K2QGF2Q5TYQHYJ5H7YA, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:32.001947612 (501h18m31.873030647s) [noise=true , address=145.79.227.28:4750 , piece_id=SYOPLUXNCR7KCMR7FX6PMICD4SDW7VLAKNWRWVC3455ESX4P4IKQ, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:32.002400785 (501h18m31.87348383s) [noise=true , address=45.91.116.8:4074 , piece_id=YDTDZNKX5LRO3V3UQHWQB64PS7OBRMIZEBIJQ4TWONL5K2KRZVWA, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:32.002666842 (501h18m31.873749877s) [noise=true , address=45.91.116.11:4146 , piece_id=X7UK2OJXVANXFXYCWRZ6M3UIYULU4DFJ4I2UESRR7PRYC6NCXJOA, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:32.003349923 (501h18m31.874432968s) [noise=true , address=77.90.31.14:4671 , piece_id=O7VFBBQLQ5FBP4UPQFELDDKB5JR4GUEFY6I62BVS4D7VCXRT7R5A, seg_pos={0 0}, upload=439960]: Uploading piece... Dec 26 12:47:33.469721230 (501h18m33.340804265s) [upload=439960]: Next returned nil segment Dec 26 12:47:33.469757930 (501h18m33.340840975s) [upload=439960]: Waiting for error group managing segments... Dec 26 12:47:33.864743802 (501h18m33.735826837s) [noise=true , address=145.79.227.29:4954 , piece_id=6BH4PB2YY6UMBZHSU7B4MS64PSFHC5SRFCNSYUWHLHKQTUKR7EDQ, seg_pos={0 0}, upload=439960]: Done uploading piece. err: <nil> Dec 26 12:47:33.878583104 (501h18m33.749666149s) [noise=true , address=145.79.227.21:4486 , piece_id=V7DWPLGDLWDYE4TJLGIFMTOLDCXEDBTPUYSWYXPFXOLG3KZDKDQA, seg_pos={0 0}, upload=439960]: Done uploading piece. err: <nil> Dec 26 12:47:33.885532327 (501h18m33.756615362s) [noise=true , address=145.79.227.28:4750 , piece_id=SYOPLUXNCR7KCMR7FX6PMICD4SDW7VLAKNWRWVC3455ESX4P4IKQ, seg_pos={0 0}, upload=439960]: Done uploading piece. err: <nil> Dec 26 12:47:33.889919020 (501h18m33.761002065s) [noise=true , address=145.79.227.43:4171 , piece_id=MDGS63GO7OPP6Y2YYCHJOZSZK25X3YTJHXDEOG4RZAF5LZ4IABZA, seg_pos={0 0}, upload=439960]: Done uploading piece. err: <nil> Dec 26 12:47:34.047203828 (501h18m33.918286873s) [noise=true , address=77.90.31.15:4708 , piece_id=PPFXUZ7QDC33NCS6PMM4Z4SWBGFYPGJPBVQYE3FLPONDFHMXLXSQ, seg_pos={0 0}, upload=439960]: Done uploading piece. err: <nil> Dec 26 12:47:34.070368728 (501h18m33.941451763s) [noise=true , address=45.91.116.4:4962 , piece_id=P5SBO3YE6QCFXRVAL62G3HC5XJDNJBPA3PMQEYLNB2U7M46IXKCA, seg_pos={0 0}, upload=439960]: Done uploading piece. err: <nil> Dec 26 12:47:34.091662365 (501h18m33.9627454s) [noise=true , address=77.90.31.16:4757 , piece_id=QOXGFQE7CP3745MSKKMAYIDMD4CEGI642K2QGF2Q5TYQHYJ5H7YA, seg_pos={0 0}, upload=439960]: Done uploading piece. err: <nil> Dec 26 12:47:34.143840831 (501h18m34.014923867s) [noise=true , address=77.90.31.19:4849 , piece_id=BB52EEC5VJKTJ7H722N7SXUMWBWAUAVFMRHMPORK3KC6QY6LMHZQ, seg_pos={0 0}, upload=439960]: Done uploading piece. err: <nil> Dec 26 12:47:34.601885600 (501h18m34.472968655s) [noise=true , address=77.90.31.14:4671 , piece_id=O7VFBBQLQ5FBP4UPQFELDDKB5JR4GUEFY6I62BVS4D7VCXRT7R5A, seg_pos={0 0}, upload=439960]: Done uploading piece. err: <nil> Dec 26 12:47:34.776497801 (501h18m34.647580836s) [noise=true , address=145.79.227.40:4069 , piece_id=ISS3ZLARB55VNPWUMFL3WXHGC3NZMKSFWPLVTPWEVZJ7EXUPF5WQ, seg_pos={0 0}, upload=439960]: Done uploading piece. err: <nil> Dec 26 12:47:34.777613185 (501h18m34.64869623s) [noise=true , address=145.79.227.44:4226 , piece_id=FQY74SGDMFVPM54JVMLG2DATBKKX4UI2JLO5C5YIUFONTAMPTJQQ, seg_pos={0 0}, upload=439960]: Done uploading piece. err: <nil> Dec 26 12:47:34.792547793 (501h18m34.663630828s) [noise=true , address=145.79.227.30:4782 , piece_id=KAEZ3CPVWDFBFEEAKZXLFCGJ7XAYYXRQV5TMXZUS2AD6CBX3HPTA, seg_pos={0 0}, upload=439960]: Done uploading piece. err: <nil> Dec 26 12:47:34.830745548 (501h18m34.701828583s) [noise=true , address=77.90.31.10:4531 , piece_id=FQ5HW32FOAV5VHVLXIFPWWG67DVCKE5OSIDSWAEDJ7FH3USONACA, seg_pos={0 0}, upload=439960]: Done uploading piece. err: <nil> Dec 26 12:47:34.951612160 (501h18m34.822695205s) [noise=true , address=145.79.227.34:4939 , piece_id=FAEDI2PHQ543A5OOP6K73ZDWCM3TPTQQBJPFEYOAGW26J7GX5S6Q, seg_pos={0 0}, upload=439960]: Done uploading piece. err: <nil> Dec 26 12:47:34.951634812 (501h18m34.822717857s) [seg_pos={0 0}, upload=439960]: Segment reached optimal threshold of 14 pieces. Dec 26 12:47:34.951856816 (501h18m34.822939851s) [noise=true , address=45.91.116.11:4146 , piece_id=X7UK2OJXVANXFXYCWRZ6M3UIYULU4DFJ4I2UESRR7PRYC6NCXJOA, seg_pos={0 0}, upload=439960]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp): protocol: expected piece hash; context canceled; EOF Dec 26 12:47:34.951864901 (501h18m34.822947946s) [noise=true , address=45.91.116.11:4146 , piece_id=X7UK2OJXVANXFXYCWRZ6M3UIYULU4DFJ4I2UESRR7PRYC6NCXJOA, seg_pos={0 0}, upload=439960]: Long tail cancellation due to optimal threshold Dec 26 12:47:34.951971644 (501h18m34.823054709s) [noise=true , address=45.91.116.8:4074 , piece_id=YDTDZNKX5LRO3V3UQHWQB64PS7OBRMIZEBIJQ4TWONL5K2KRZVWA, seg_pos={0 0}, upload=439960]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12whTD1Kvu4T7ANNtkSvDB66uTjVgfPTK9TAzExgvwJYgZ3zSFG): protocol: expected piece hash; context canceled; EOF Dec 26 12:47:34.951991683 (501h18m34.823074718s) [noise=true , address=45.91.116.8:4074 , piece_id=YDTDZNKX5LRO3V3UQHWQB64PS7OBRMIZEBIJQ4TWONL5K2KRZVWA, seg_pos={0 0}, upload=439960]: Long tail cancellation due to optimal threshold Dec 26 12:47:34.951995770 (501h18m34.823078805s) [noise=true , address=45.91.116.6:4993 , piece_id=PKE5HHLE4GKQ34O3I2FONBGICCHP7TCMTLUFQIC4PIXOU5L5MBTQ, seg_pos={0 0}, upload=439960]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XP9gxzzBfzvzxZTatrKN9nLFvxsBuAioHxgNhSHmJYU8NAJfc): protocol: expected piece hash; context canceled; EOF Dec 26 12:47:34.952004647 (501h18m34.823087682s) [noise=true , address=45.91.116.6:4993 , piece_id=PKE5HHLE4GKQ34O3I2FONBGICCHP7TCMTLUFQIC4PIXOU5L5MBTQ, seg_pos={0 0}, upload=439960]: Long tail cancellation due to optimal threshold Dec 26 12:47:34.952136378 (501h18m34.823219413s) [noise=true , address=77.90.31.11:4562 , piece_id=4D42T5I4SG3PZMQNR6NYWA2OQYYRCKTYL3PP33SKHZYQSE3WO5NQ, seg_pos={0 0}, upload=439960]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQ4wGnLgKifp7pWWtKGtCom5K7fGceJHy2SyLGvUemZx4H4r): protocol: expected piece hash; context canceled; EOF Dec 26 12:47:34.952143882 (501h18m34.823226917s) [noise=true , address=77.90.31.11:4562 , piece_id=4D42T5I4SG3PZMQNR6NYWA2OQYYRCKTYL3PP33SKHZYQSE3WO5NQ, seg_pos={0 0}, upload=439960]: Long tail cancellation due to optimal threshold Dec 26 12:47:34.952185611 (501h18m34.823268646s) [seg_pos={0 0}, upload=439960]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 12:47:34.952264482 (501h18m34.823347517s) [upload=439960]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:47:34.952270033 (501h18m34.823353068s) [upload=439960]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:47:35.254157018 (501h18m35.125240084s) [upload=439960]: Done uploading segments... Dec 26 12:47:35.254181404 (501h18m35.125264439s) [upload=439960]: Upload finished. err: <nil> Duration: 1m0.002957822s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [..X..X......X.......XXX...X.....X.......X.XX......XXX..XXXXX] duration: 5.00114842s trace: https://jaeger.dev.storj.io/trace/7b221e89e77876a7 FAIL (2025-12-26 12:48:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001172877s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X....X..X...X..XX.....XXXX.....XXX....XX....X..X......X.X..] duration: 5.956830037s trace: https://jaeger.dev.storj.io/trace/123468ab5ee87146 FAIL (2025-12-26 12:48:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: DIAOOLPMEL6KYV7TEYS6LK2AAVKC6JBSNQ5EGRKLGVJGZVUWLDOQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: WOECGYFHQVMDZFD7GH5RMOPOPM5TC3HJXMZW4VWGFN5XQ2YZZ2YA) context deadline exceeded ======LOGS====== Dec 26 12:46:57.034323814 (501h17m56.905406849s) [upload=439948]: Starting upload Dec 26 12:46:57.034395801 (501h17m56.905478846s) [upload=439948]: Done starting upload Dec 26 12:46:57.034505450 (501h17m56.905588485s) [upload=439948]: Uploading segments... Dec 26 12:46:57.034728726 (501h17m56.905811761s) [upload=439948]: Got next segment. Inline: false Dec 26 12:46:57.034761258 (501h17m56.905844303s) [upload=439948]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:46:57.034766157 (501h17m56.905849192s) [upload=439948]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:46:57.276786817 (501h17m57.147869882s) [seg_pos={0 0}, upload=439948]: Begin upload segment... Dec 26 12:46:57.276850186 (501h17m57.147933231s) [seg_pos={0 0}, upload=439948]: Done begin upload segment. Dec 26 12:46:57.276866678 (501h17m57.147949723s) [noise=true , address=45.91.116.5:4298 , piece_id=PJWDLHJONXT4EL33Z5IGDBGE3VQS2NIGZUS2W4SU7CBYXNZTHHJQ, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:57.276892758 (501h17m57.147975803s) [noise=true , address=145.79.227.42:4122 , piece_id=JX6ANSA3Y4EGKWLXJA6TYAHZOQLCQQKG2PSZRJV4TTL26UJYW56Q, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:57.277018868 (501h17m57.148101903s) [noise=true , address=145.79.227.52:4445 , piece_id=IARUS4O4BYIOHUVZKAIC26OBYOPK3SWP37ELGIDGTK2TJ25G5TIA, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:57.277589275 (501h17m57.14867233s) [noise=true , address=145.79.227.4:4186 , piece_id=IN7SP2KFMKXMTSHVMM2WMDRRF3J3KH6QOALXVGMNKRJBAH6PWRBQ, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:57.278238670 (501h17m57.149321716s) [noise=true , address=145.79.227.15:4874 , piece_id=3VDCBYBPFR3TB4TIAX7MSKZ6LPFLEU3RWAZOEJUDUCFL4D3ZIKFA, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:57.278938673 (501h17m57.150021738s) [noise=true , address=145.79.227.35:4482 , piece_id=OZD4BKKQGKGSB5XRRSAUEXJ372Y6MFPAFFN6YCUTCY4RMDL33QNQ, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:57.279520872 (501h17m57.150603937s) [noise=true , address=45.91.116.4:4937 , piece_id=KWHXM65G5Z2U7TIXUWAAYNBMOILLY4OUCJG56HERDAF72B2JQAUA, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:57.280828863 (501h17m57.151911908s) [noise=true , address=145.79.227.48:4301 , piece_id=Q6WQHO6WJE2CXZLC3LH672GMI5K56WUFWTTRNO4JN6FOMXFR767A, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:57.281637162 (501h17m57.152720207s) [noise=true , address=45.91.116.14:4189 , piece_id=R7CFOFGMOTHN77YISSVKDVDBVU3DQMKB7DZN45CL3ZPHZHK7YNWQ, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:57.282119591 (501h17m57.153202626s) [noise=true , address=145.79.227.41:4086 , piece_id=2ZYEAYTAERUKHFIBT7CQXGJHRZDYKCOXE4HJTSFKEI772R4LMEEA, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:57.283499348 (501h17m57.154582383s) [noise=true , address=145.79.227.6:4259 , piece_id=ON73ZKAWQODJ6OL3ZJJINLLASN6L6P7D5OLZAYO75EABEDU2ESHA, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:57.284304111 (501h17m57.155387146s) [noise=true , address=145.79.227.51:4410 , piece_id=XKVM33NB6BVGUZ6ICZQQTCRCQOQB3UZJ7JZA7BJ6GHFH6HDIMCOA, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:57.284810935 (501h17m57.15589398s) [noise=true , address=45.91.116.11:4117 , piece_id=DIAOOLPMEL6KYV7TEYS6LK2AAVKC6JBSNQ5EGRKLGVJGZVUWLDOQ, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:57.286270735 (501h17m57.15735378s) [noise=true , address=145.79.227.13:4803 , piece_id=IDI26I6LO4YFBSRWXT55P6WO2GRNJWS4UIAEXJRDQN4AKZGRKINA, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:57.287093091 (501h17m57.158176147s) [noise=true , address=145.79.227.11:4730 , piece_id=MXGLU2HP6GMORSFVX6SA2FBYV2RDXZNXIQYWBSWP4GTWN6B64BKQ, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:57.287670771 (501h17m57.158753806s) [noise=true , address=77.90.31.14:4663 , piece_id=WOECGYFHQVMDZFD7GH5RMOPOPM5TC3HJXMZW4VWGFN5XQ2YZZ2YA, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:58.926706422 (501h17m58.797789468s) [upload=439948]: Next returned nil segment Dec 26 12:46:58.926719026 (501h17m58.797802071s) [upload=439948]: Waiting for error group managing segments... Dec 26 12:46:59.313840074 (501h17m59.184923109s) [noise=true , address=145.79.227.13:4803 , piece_id=IDI26I6LO4YFBSRWXT55P6WO2GRNJWS4UIAEXJRDQN4AKZGRKINA, seg_pos={0 0}, upload=439948]: Done uploading piece. err: <nil> Dec 26 12:46:59.316492815 (501h17m59.18757586s) [noise=true , address=145.79.227.11:4730 , piece_id=MXGLU2HP6GMORSFVX6SA2FBYV2RDXZNXIQYWBSWP4GTWN6B64BKQ, seg_pos={0 0}, upload=439948]: Done uploading piece. err: <nil> Dec 26 12:46:59.317120390 (501h17m59.188203435s) [noise=true , address=145.79.227.41:4086 , piece_id=2ZYEAYTAERUKHFIBT7CQXGJHRZDYKCOXE4HJTSFKEI772R4LMEEA, seg_pos={0 0}, upload=439948]: Done uploading piece. err: <nil> Dec 26 12:46:59.317949970 (501h17m59.189033005s) [noise=true , address=145.79.227.51:4410 , piece_id=XKVM33NB6BVGUZ6ICZQQTCRCQOQB3UZJ7JZA7BJ6GHFH6HDIMCOA, seg_pos={0 0}, upload=439948]: Done uploading piece. err: <nil> Dec 26 12:46:59.318355341 (501h17m59.189438376s) [noise=true , address=145.79.227.48:4301 , piece_id=Q6WQHO6WJE2CXZLC3LH672GMI5K56WUFWTTRNO4JN6FOMXFR767A, seg_pos={0 0}, upload=439948]: Done uploading piece. err: <nil> Dec 26 12:46:59.319021910 (501h17m59.190104946s) [noise=true , address=145.79.227.4:4186 , piece_id=IN7SP2KFMKXMTSHVMM2WMDRRF3J3KH6QOALXVGMNKRJBAH6PWRBQ, seg_pos={0 0}, upload=439948]: Done uploading piece. err: <nil> Dec 26 12:46:59.319242561 (501h17m59.190325606s) [noise=true , address=145.79.227.35:4482 , piece_id=OZD4BKKQGKGSB5XRRSAUEXJ372Y6MFPAFFN6YCUTCY4RMDL33QNQ, seg_pos={0 0}, upload=439948]: Done uploading piece. err: <nil> Dec 26 12:46:59.320056961 (501h17m59.191140006s) [noise=true , address=145.79.227.52:4445 , piece_id=IARUS4O4BYIOHUVZKAIC26OBYOPK3SWP37ELGIDGTK2TJ25G5TIA, seg_pos={0 0}, upload=439948]: Done uploading piece. err: <nil> Dec 26 12:46:59.321182024 (501h17m59.192265069s) [noise=true , address=145.79.227.6:4259 , piece_id=ON73ZKAWQODJ6OL3ZJJINLLASN6L6P7D5OLZAYO75EABEDU2ESHA, seg_pos={0 0}, upload=439948]: Done uploading piece. err: <nil> Dec 26 12:46:59.327001265 (501h17m59.19808431s) [noise=true , address=145.79.227.15:4874 , piece_id=3VDCBYBPFR3TB4TIAX7MSKZ6LPFLEU3RWAZOEJUDUCFL4D3ZIKFA, seg_pos={0 0}, upload=439948]: Done uploading piece. err: <nil> Dec 26 12:46:59.333334554 (501h17m59.2044176s) [noise=true , address=145.79.227.42:4122 , piece_id=JX6ANSA3Y4EGKWLXJA6TYAHZOQLCQQKG2PSZRJV4TTL26UJYW56Q, seg_pos={0 0}, upload=439948]: Done uploading piece. err: <nil> Dec 26 12:46:59.719362208 (501h17m59.590445254s) [noise=true , address=77.90.31.14:4663 , piece_id=WOECGYFHQVMDZFD7GH5RMOPOPM5TC3HJXMZW4VWGFN5XQ2YZZ2YA, seg_pos={0 0}, upload=439948]: Done uploading piece. err: <nil> Dec 26 12:47:00.251543865 (501h18m0.12262691s) [noise=true , address=45.91.116.11:4117 , piece_id=DIAOOLPMEL6KYV7TEYS6LK2AAVKC6JBSNQ5EGRKLGVJGZVUWLDOQ, seg_pos={0 0}, upload=439948]: Done uploading piece. err: <nil> Dec 26 12:47:00.251571087 (501h18m0.122654132s) [seg_pos={0 0}, upload=439948]: Segment reached optimal threshold of 13 pieces. Dec 26 12:47:00.251840671 (501h18m0.122923716s) [noise=true , address=45.91.116.5:4298 , piece_id=PJWDLHJONXT4EL33Z5IGDBGE3VQS2NIGZUS2W4SU7CBYXNZTHHJQ, seg_pos={0 0}, upload=439948]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 26 12:47:00.251857062 (501h18m0.122940097s) [noise=true , address=45.91.116.5:4298 , piece_id=PJWDLHJONXT4EL33Z5IGDBGE3VQS2NIGZUS2W4SU7CBYXNZTHHJQ, seg_pos={0 0}, upload=439948]: Long tail cancellation due to optimal threshold Dec 26 12:47:00.251876309 (501h18m0.122959354s) [noise=true , address=45.91.116.14:4189 , piece_id=R7CFOFGMOTHN77YISSVKDVDBVU3DQMKB7DZN45CL3ZPHZHK7YNWQ, seg_pos={0 0}, upload=439948]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 26 12:47:00.251889594 (501h18m0.122972629s) [noise=true , address=45.91.116.14:4189 , piece_id=R7CFOFGMOTHN77YISSVKDVDBVU3DQMKB7DZN45CL3ZPHZHK7YNWQ, seg_pos={0 0}, upload=439948]: Long tail cancellation due to optimal threshold Dec 26 12:47:00.252244209 (501h18m0.123327244s) [noise=true , address=45.91.116.4:4937 , piece_id=KWHXM65G5Z2U7TIXUWAAYNBMOILLY4OUCJG56HERDAF72B2JQAUA, seg_pos={0 0}, upload=439948]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 12:47:00.252251072 (501h18m0.123334117s) [noise=true , address=45.91.116.4:4937 , piece_id=KWHXM65G5Z2U7TIXUWAAYNBMOILLY4OUCJG56HERDAF72B2JQAUA, seg_pos={0 0}, upload=439948]: Long tail cancellation due to optimal threshold Dec 26 12:47:00.252271341 (501h18m0.123354376s) [seg_pos={0 0}, upload=439948]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:47:00.252348187 (501h18m0.123431232s) [upload=439948]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:47:00.252353377 (501h18m0.123436422s) [upload=439948]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:47:00.533600250 (501h18m0.404683315s) [upload=439948]: Done uploading segments... Dec 26 12:47:00.533628233 (501h18m0.404711278s) [upload=439948]: Upload finished. err: <nil> Duration: 5.95686369s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [...X..X......X.......XXX...X.....X.......X.XX......XXX..XXXX] duration: 5.001259083s trace: https://jaeger.dev.storj.io/trace/3107291cf60ad6ba FAIL (2025-12-26 12:47:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001285423s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X.X....X..X...X..XX.....XXXX.....XXX....XX....X..X......X.X.] duration: 5.839832252s trace: https://jaeger.dev.storj.io/trace/123468ab5ee87146 FAIL (2025-12-26 12:47:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: DIAOOLPMEL6KYV7TEYS6LK2AAVKC6JBSNQ5EGRKLGVJGZVUWLDOQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: WOECGYFHQVMDZFD7GH5RMOPOPM5TC3HJXMZW4VWGFN5XQ2YZZ2YA) context deadline exceeded ======LOGS====== Dec 26 12:46:57.034323814 (501h17m56.905406849s) [upload=439948]: Starting upload Dec 26 12:46:57.034395801 (501h17m56.905478846s) [upload=439948]: Done starting upload Dec 26 12:46:57.034505450 (501h17m56.905588485s) [upload=439948]: Uploading segments... Dec 26 12:46:57.034728726 (501h17m56.905811761s) [upload=439948]: Got next segment. Inline: false Dec 26 12:46:57.034761258 (501h17m56.905844303s) [upload=439948]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:46:57.034766157 (501h17m56.905849192s) [upload=439948]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:46:57.276786817 (501h17m57.147869882s) [seg_pos={0 0}, upload=439948]: Begin upload segment... Dec 26 12:46:57.276850186 (501h17m57.147933231s) [seg_pos={0 0}, upload=439948]: Done begin upload segment. Dec 26 12:46:57.276866678 (501h17m57.147949723s) [noise=true , address=45.91.116.5:4298 , piece_id=PJWDLHJONXT4EL33Z5IGDBGE3VQS2NIGZUS2W4SU7CBYXNZTHHJQ, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:57.276892758 (501h17m57.147975803s) [noise=true , address=145.79.227.42:4122 , piece_id=JX6ANSA3Y4EGKWLXJA6TYAHZOQLCQQKG2PSZRJV4TTL26UJYW56Q, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:57.277018868 (501h17m57.148101903s) [noise=true , address=145.79.227.52:4445 , piece_id=IARUS4O4BYIOHUVZKAIC26OBYOPK3SWP37ELGIDGTK2TJ25G5TIA, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:57.277589275 (501h17m57.14867233s) [noise=true , address=145.79.227.4:4186 , piece_id=IN7SP2KFMKXMTSHVMM2WMDRRF3J3KH6QOALXVGMNKRJBAH6PWRBQ, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:57.278238670 (501h17m57.149321716s) [noise=true , address=145.79.227.15:4874 , piece_id=3VDCBYBPFR3TB4TIAX7MSKZ6LPFLEU3RWAZOEJUDUCFL4D3ZIKFA, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:57.278938673 (501h17m57.150021738s) [noise=true , address=145.79.227.35:4482 , piece_id=OZD4BKKQGKGSB5XRRSAUEXJ372Y6MFPAFFN6YCUTCY4RMDL33QNQ, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:57.279520872 (501h17m57.150603937s) [noise=true , address=45.91.116.4:4937 , piece_id=KWHXM65G5Z2U7TIXUWAAYNBMOILLY4OUCJG56HERDAF72B2JQAUA, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:57.280828863 (501h17m57.151911908s) [noise=true , address=145.79.227.48:4301 , piece_id=Q6WQHO6WJE2CXZLC3LH672GMI5K56WUFWTTRNO4JN6FOMXFR767A, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:57.281637162 (501h17m57.152720207s) [noise=true , address=45.91.116.14:4189 , piece_id=R7CFOFGMOTHN77YISSVKDVDBVU3DQMKB7DZN45CL3ZPHZHK7YNWQ, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:57.282119591 (501h17m57.153202626s) [noise=true , address=145.79.227.41:4086 , piece_id=2ZYEAYTAERUKHFIBT7CQXGJHRZDYKCOXE4HJTSFKEI772R4LMEEA, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:57.283499348 (501h17m57.154582383s) [noise=true , address=145.79.227.6:4259 , piece_id=ON73ZKAWQODJ6OL3ZJJINLLASN6L6P7D5OLZAYO75EABEDU2ESHA, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:57.284304111 (501h17m57.155387146s) [noise=true , address=145.79.227.51:4410 , piece_id=XKVM33NB6BVGUZ6ICZQQTCRCQOQB3UZJ7JZA7BJ6GHFH6HDIMCOA, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:57.284810935 (501h17m57.15589398s) [noise=true , address=45.91.116.11:4117 , piece_id=DIAOOLPMEL6KYV7TEYS6LK2AAVKC6JBSNQ5EGRKLGVJGZVUWLDOQ, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:57.286270735 (501h17m57.15735378s) [noise=true , address=145.79.227.13:4803 , piece_id=IDI26I6LO4YFBSRWXT55P6WO2GRNJWS4UIAEXJRDQN4AKZGRKINA, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:57.287093091 (501h17m57.158176147s) [noise=true , address=145.79.227.11:4730 , piece_id=MXGLU2HP6GMORSFVX6SA2FBYV2RDXZNXIQYWBSWP4GTWN6B64BKQ, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:57.287670771 (501h17m57.158753806s) [noise=true , address=77.90.31.14:4663 , piece_id=WOECGYFHQVMDZFD7GH5RMOPOPM5TC3HJXMZW4VWGFN5XQ2YZZ2YA, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:58.926706422 (501h17m58.797789468s) [upload=439948]: Next returned nil segment Dec 26 12:46:58.926719026 (501h17m58.797802071s) [upload=439948]: Waiting for error group managing segments... Dec 26 12:46:59.313840074 (501h17m59.184923109s) [noise=true , address=145.79.227.13:4803 , piece_id=IDI26I6LO4YFBSRWXT55P6WO2GRNJWS4UIAEXJRDQN4AKZGRKINA, seg_pos={0 0}, upload=439948]: Done uploading piece. err: <nil> Dec 26 12:46:59.316492815 (501h17m59.18757586s) [noise=true , address=145.79.227.11:4730 , piece_id=MXGLU2HP6GMORSFVX6SA2FBYV2RDXZNXIQYWBSWP4GTWN6B64BKQ, seg_pos={0 0}, upload=439948]: Done uploading piece. err: <nil> Dec 26 12:46:59.317120390 (501h17m59.188203435s) [noise=true , address=145.79.227.41:4086 , piece_id=2ZYEAYTAERUKHFIBT7CQXGJHRZDYKCOXE4HJTSFKEI772R4LMEEA, seg_pos={0 0}, upload=439948]: Done uploading piece. err: <nil> Dec 26 12:46:59.317949970 (501h17m59.189033005s) [noise=true , address=145.79.227.51:4410 , piece_id=XKVM33NB6BVGUZ6ICZQQTCRCQOQB3UZJ7JZA7BJ6GHFH6HDIMCOA, seg_pos={0 0}, upload=439948]: Done uploading piece. err: <nil> Dec 26 12:46:59.318355341 (501h17m59.189438376s) [noise=true , address=145.79.227.48:4301 , piece_id=Q6WQHO6WJE2CXZLC3LH672GMI5K56WUFWTTRNO4JN6FOMXFR767A, seg_pos={0 0}, upload=439948]: Done uploading piece. err: <nil> Dec 26 12:46:59.319021910 (501h17m59.190104946s) [noise=true , address=145.79.227.4:4186 , piece_id=IN7SP2KFMKXMTSHVMM2WMDRRF3J3KH6QOALXVGMNKRJBAH6PWRBQ, seg_pos={0 0}, upload=439948]: Done uploading piece. err: <nil> Dec 26 12:46:59.319242561 (501h17m59.190325606s) [noise=true , address=145.79.227.35:4482 , piece_id=OZD4BKKQGKGSB5XRRSAUEXJ372Y6MFPAFFN6YCUTCY4RMDL33QNQ, seg_pos={0 0}, upload=439948]: Done uploading piece. err: <nil> Dec 26 12:46:59.320056961 (501h17m59.191140006s) [noise=true , address=145.79.227.52:4445 , piece_id=IARUS4O4BYIOHUVZKAIC26OBYOPK3SWP37ELGIDGTK2TJ25G5TIA, seg_pos={0 0}, upload=439948]: Done uploading piece. err: <nil> Dec 26 12:46:59.321182024 (501h17m59.192265069s) [noise=true , address=145.79.227.6:4259 , piece_id=ON73ZKAWQODJ6OL3ZJJINLLASN6L6P7D5OLZAYO75EABEDU2ESHA, seg_pos={0 0}, upload=439948]: Done uploading piece. err: <nil> Dec 26 12:46:59.327001265 (501h17m59.19808431s) [noise=true , address=145.79.227.15:4874 , piece_id=3VDCBYBPFR3TB4TIAX7MSKZ6LPFLEU3RWAZOEJUDUCFL4D3ZIKFA, seg_pos={0 0}, upload=439948]: Done uploading piece. err: <nil> Dec 26 12:46:59.333334554 (501h17m59.2044176s) [noise=true , address=145.79.227.42:4122 , piece_id=JX6ANSA3Y4EGKWLXJA6TYAHZOQLCQQKG2PSZRJV4TTL26UJYW56Q, seg_pos={0 0}, upload=439948]: Done uploading piece. err: <nil> Dec 26 12:46:59.719362208 (501h17m59.590445254s) [noise=true , address=77.90.31.14:4663 , piece_id=WOECGYFHQVMDZFD7GH5RMOPOPM5TC3HJXMZW4VWGFN5XQ2YZZ2YA, seg_pos={0 0}, upload=439948]: Done uploading piece. err: <nil> Dec 26 12:47:00.251543865 (501h18m0.12262691s) [noise=true , address=45.91.116.11:4117 , piece_id=DIAOOLPMEL6KYV7TEYS6LK2AAVKC6JBSNQ5EGRKLGVJGZVUWLDOQ, seg_pos={0 0}, upload=439948]: Done uploading piece. err: <nil> Dec 26 12:47:00.251571087 (501h18m0.122654132s) [seg_pos={0 0}, upload=439948]: Segment reached optimal threshold of 13 pieces. Dec 26 12:47:00.251840671 (501h18m0.122923716s) [noise=true , address=45.91.116.5:4298 , piece_id=PJWDLHJONXT4EL33Z5IGDBGE3VQS2NIGZUS2W4SU7CBYXNZTHHJQ, seg_pos={0 0}, upload=439948]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 26 12:47:00.251857062 (501h18m0.122940097s) [noise=true , address=45.91.116.5:4298 , piece_id=PJWDLHJONXT4EL33Z5IGDBGE3VQS2NIGZUS2W4SU7CBYXNZTHHJQ, seg_pos={0 0}, upload=439948]: Long tail cancellation due to optimal threshold Dec 26 12:47:00.251876309 (501h18m0.122959354s) [noise=true , address=45.91.116.14:4189 , piece_id=R7CFOFGMOTHN77YISSVKDVDBVU3DQMKB7DZN45CL3ZPHZHK7YNWQ, seg_pos={0 0}, upload=439948]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 26 12:47:00.251889594 (501h18m0.122972629s) [noise=true , address=45.91.116.14:4189 , piece_id=R7CFOFGMOTHN77YISSVKDVDBVU3DQMKB7DZN45CL3ZPHZHK7YNWQ, seg_pos={0 0}, upload=439948]: Long tail cancellation due to optimal threshold Dec 26 12:47:00.252244209 (501h18m0.123327244s) [noise=true , address=45.91.116.4:4937 , piece_id=KWHXM65G5Z2U7TIXUWAAYNBMOILLY4OUCJG56HERDAF72B2JQAUA, seg_pos={0 0}, upload=439948]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 12:47:00.252251072 (501h18m0.123334117s) [noise=true , address=45.91.116.4:4937 , piece_id=KWHXM65G5Z2U7TIXUWAAYNBMOILLY4OUCJG56HERDAF72B2JQAUA, seg_pos={0 0}, upload=439948]: Long tail cancellation due to optimal threshold Dec 26 12:47:00.252271341 (501h18m0.123354376s) [seg_pos={0 0}, upload=439948]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:47:00.252348187 (501h18m0.123431232s) [upload=439948]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:47:00.252353377 (501h18m0.123436422s) [upload=439948]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:47:00.533600250 (501h18m0.404683315s) [upload=439948]: Done uploading segments... Dec 26 12:47:00.533628233 (501h18m0.404711278s) [upload=439948]: Upload finished. err: <nil> Duration: 5.839858272s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X.X....X..X...X..XX.....XXXX.....XXX....XX....X..X......X.X] duration: 30.003250934s trace: https://jaeger.dev.storj.io/trace/123468ab5ee87146 FAIL (2025-12-26 12:47:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: DIAOOLPMEL6KYV7TEYS6LK2AAVKC6JBSNQ5EGRKLGVJGZVUWLDOQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: WOECGYFHQVMDZFD7GH5RMOPOPM5TC3HJXMZW4VWGFN5XQ2YZZ2YA) context deadline exceeded ======LOGS====== Dec 26 12:46:57.034323814 (501h17m56.905406849s) [upload=439948]: Starting upload Dec 26 12:46:57.034395801 (501h17m56.905478846s) [upload=439948]: Done starting upload Dec 26 12:46:57.034505450 (501h17m56.905588485s) [upload=439948]: Uploading segments... Dec 26 12:46:57.034728726 (501h17m56.905811761s) [upload=439948]: Got next segment. Inline: false Dec 26 12:46:57.034761258 (501h17m56.905844303s) [upload=439948]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:46:57.034766157 (501h17m56.905849192s) [upload=439948]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:46:57.276786817 (501h17m57.147869882s) [seg_pos={0 0}, upload=439948]: Begin upload segment... Dec 26 12:46:57.276850186 (501h17m57.147933231s) [seg_pos={0 0}, upload=439948]: Done begin upload segment. Dec 26 12:46:57.276866678 (501h17m57.147949723s) [noise=true , address=45.91.116.5:4298 , piece_id=PJWDLHJONXT4EL33Z5IGDBGE3VQS2NIGZUS2W4SU7CBYXNZTHHJQ, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:57.276892758 (501h17m57.147975803s) [noise=true , address=145.79.227.42:4122 , piece_id=JX6ANSA3Y4EGKWLXJA6TYAHZOQLCQQKG2PSZRJV4TTL26UJYW56Q, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:57.277018868 (501h17m57.148101903s) [noise=true , address=145.79.227.52:4445 , piece_id=IARUS4O4BYIOHUVZKAIC26OBYOPK3SWP37ELGIDGTK2TJ25G5TIA, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:57.277589275 (501h17m57.14867233s) [noise=true , address=145.79.227.4:4186 , piece_id=IN7SP2KFMKXMTSHVMM2WMDRRF3J3KH6QOALXVGMNKRJBAH6PWRBQ, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:57.278238670 (501h17m57.149321716s) [noise=true , address=145.79.227.15:4874 , piece_id=3VDCBYBPFR3TB4TIAX7MSKZ6LPFLEU3RWAZOEJUDUCFL4D3ZIKFA, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:57.278938673 (501h17m57.150021738s) [noise=true , address=145.79.227.35:4482 , piece_id=OZD4BKKQGKGSB5XRRSAUEXJ372Y6MFPAFFN6YCUTCY4RMDL33QNQ, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:57.279520872 (501h17m57.150603937s) [noise=true , address=45.91.116.4:4937 , piece_id=KWHXM65G5Z2U7TIXUWAAYNBMOILLY4OUCJG56HERDAF72B2JQAUA, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:57.280828863 (501h17m57.151911908s) [noise=true , address=145.79.227.48:4301 , piece_id=Q6WQHO6WJE2CXZLC3LH672GMI5K56WUFWTTRNO4JN6FOMXFR767A, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:57.281637162 (501h17m57.152720207s) [noise=true , address=45.91.116.14:4189 , piece_id=R7CFOFGMOTHN77YISSVKDVDBVU3DQMKB7DZN45CL3ZPHZHK7YNWQ, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:57.282119591 (501h17m57.153202626s) [noise=true , address=145.79.227.41:4086 , piece_id=2ZYEAYTAERUKHFIBT7CQXGJHRZDYKCOXE4HJTSFKEI772R4LMEEA, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:57.283499348 (501h17m57.154582383s) [noise=true , address=145.79.227.6:4259 , piece_id=ON73ZKAWQODJ6OL3ZJJINLLASN6L6P7D5OLZAYO75EABEDU2ESHA, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:57.284304111 (501h17m57.155387146s) [noise=true , address=145.79.227.51:4410 , piece_id=XKVM33NB6BVGUZ6ICZQQTCRCQOQB3UZJ7JZA7BJ6GHFH6HDIMCOA, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:57.284810935 (501h17m57.15589398s) [noise=true , address=45.91.116.11:4117 , piece_id=DIAOOLPMEL6KYV7TEYS6LK2AAVKC6JBSNQ5EGRKLGVJGZVUWLDOQ, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:57.286270735 (501h17m57.15735378s) [noise=true , address=145.79.227.13:4803 , piece_id=IDI26I6LO4YFBSRWXT55P6WO2GRNJWS4UIAEXJRDQN4AKZGRKINA, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:57.287093091 (501h17m57.158176147s) [noise=true , address=145.79.227.11:4730 , piece_id=MXGLU2HP6GMORSFVX6SA2FBYV2RDXZNXIQYWBSWP4GTWN6B64BKQ, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:57.287670771 (501h17m57.158753806s) [noise=true , address=77.90.31.14:4663 , piece_id=WOECGYFHQVMDZFD7GH5RMOPOPM5TC3HJXMZW4VWGFN5XQ2YZZ2YA, seg_pos={0 0}, upload=439948]: Uploading piece... Dec 26 12:46:58.926706422 (501h17m58.797789468s) [upload=439948]: Next returned nil segment Dec 26 12:46:58.926719026 (501h17m58.797802071s) [upload=439948]: Waiting for error group managing segments... Dec 26 12:46:59.313840074 (501h17m59.184923109s) [noise=true , address=145.79.227.13:4803 , piece_id=IDI26I6LO4YFBSRWXT55P6WO2GRNJWS4UIAEXJRDQN4AKZGRKINA, seg_pos={0 0}, upload=439948]: Done uploading piece. err: <nil> Dec 26 12:46:59.316492815 (501h17m59.18757586s) [noise=true , address=145.79.227.11:4730 , piece_id=MXGLU2HP6GMORSFVX6SA2FBYV2RDXZNXIQYWBSWP4GTWN6B64BKQ, seg_pos={0 0}, upload=439948]: Done uploading piece. err: <nil> Dec 26 12:46:59.317120390 (501h17m59.188203435s) [noise=true , address=145.79.227.41:4086 , piece_id=2ZYEAYTAERUKHFIBT7CQXGJHRZDYKCOXE4HJTSFKEI772R4LMEEA, seg_pos={0 0}, upload=439948]: Done uploading piece. err: <nil> Dec 26 12:46:59.317949970 (501h17m59.189033005s) [noise=true , address=145.79.227.51:4410 , piece_id=XKVM33NB6BVGUZ6ICZQQTCRCQOQB3UZJ7JZA7BJ6GHFH6HDIMCOA, seg_pos={0 0}, upload=439948]: Done uploading piece. err: <nil> Dec 26 12:46:59.318355341 (501h17m59.189438376s) [noise=true , address=145.79.227.48:4301 , piece_id=Q6WQHO6WJE2CXZLC3LH672GMI5K56WUFWTTRNO4JN6FOMXFR767A, seg_pos={0 0}, upload=439948]: Done uploading piece. err: <nil> Dec 26 12:46:59.319021910 (501h17m59.190104946s) [noise=true , address=145.79.227.4:4186 , piece_id=IN7SP2KFMKXMTSHVMM2WMDRRF3J3KH6QOALXVGMNKRJBAH6PWRBQ, seg_pos={0 0}, upload=439948]: Done uploading piece. err: <nil> Dec 26 12:46:59.319242561 (501h17m59.190325606s) [noise=true , address=145.79.227.35:4482 , piece_id=OZD4BKKQGKGSB5XRRSAUEXJ372Y6MFPAFFN6YCUTCY4RMDL33QNQ, seg_pos={0 0}, upload=439948]: Done uploading piece. err: <nil> Dec 26 12:46:59.320056961 (501h17m59.191140006s) [noise=true , address=145.79.227.52:4445 , piece_id=IARUS4O4BYIOHUVZKAIC26OBYOPK3SWP37ELGIDGTK2TJ25G5TIA, seg_pos={0 0}, upload=439948]: Done uploading piece. err: <nil> Dec 26 12:46:59.321182024 (501h17m59.192265069s) [noise=true , address=145.79.227.6:4259 , piece_id=ON73ZKAWQODJ6OL3ZJJINLLASN6L6P7D5OLZAYO75EABEDU2ESHA, seg_pos={0 0}, upload=439948]: Done uploading piece. err: <nil> Dec 26 12:46:59.327001265 (501h17m59.19808431s) [noise=true , address=145.79.227.15:4874 , piece_id=3VDCBYBPFR3TB4TIAX7MSKZ6LPFLEU3RWAZOEJUDUCFL4D3ZIKFA, seg_pos={0 0}, upload=439948]: Done uploading piece. err: <nil> Dec 26 12:46:59.333334554 (501h17m59.2044176s) [noise=true , address=145.79.227.42:4122 , piece_id=JX6ANSA3Y4EGKWLXJA6TYAHZOQLCQQKG2PSZRJV4TTL26UJYW56Q, seg_pos={0 0}, upload=439948]: Done uploading piece. err: <nil> Dec 26 12:46:59.719362208 (501h17m59.590445254s) [noise=true , address=77.90.31.14:4663 , piece_id=WOECGYFHQVMDZFD7GH5RMOPOPM5TC3HJXMZW4VWGFN5XQ2YZZ2YA, seg_pos={0 0}, upload=439948]: Done uploading piece. err: <nil> Dec 26 12:47:00.251543865 (501h18m0.12262691s) [noise=true , address=45.91.116.11:4117 , piece_id=DIAOOLPMEL6KYV7TEYS6LK2AAVKC6JBSNQ5EGRKLGVJGZVUWLDOQ, seg_pos={0 0}, upload=439948]: Done uploading piece. err: <nil> Dec 26 12:47:00.251571087 (501h18m0.122654132s) [seg_pos={0 0}, upload=439948]: Segment reached optimal threshold of 13 pieces. Dec 26 12:47:00.251840671 (501h18m0.122923716s) [noise=true , address=45.91.116.5:4298 , piece_id=PJWDLHJONXT4EL33Z5IGDBGE3VQS2NIGZUS2W4SU7CBYXNZTHHJQ, seg_pos={0 0}, upload=439948]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 26 12:47:00.251857062 (501h18m0.122940097s) [noise=true , address=45.91.116.5:4298 , piece_id=PJWDLHJONXT4EL33Z5IGDBGE3VQS2NIGZUS2W4SU7CBYXNZTHHJQ, seg_pos={0 0}, upload=439948]: Long tail cancellation due to optimal threshold Dec 26 12:47:00.251876309 (501h18m0.122959354s) [noise=true , address=45.91.116.14:4189 , piece_id=R7CFOFGMOTHN77YISSVKDVDBVU3DQMKB7DZN45CL3ZPHZHK7YNWQ, seg_pos={0 0}, upload=439948]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 26 12:47:00.251889594 (501h18m0.122972629s) [noise=true , address=45.91.116.14:4189 , piece_id=R7CFOFGMOTHN77YISSVKDVDBVU3DQMKB7DZN45CL3ZPHZHK7YNWQ, seg_pos={0 0}, upload=439948]: Long tail cancellation due to optimal threshold Dec 26 12:47:00.252244209 (501h18m0.123327244s) [noise=true , address=45.91.116.4:4937 , piece_id=KWHXM65G5Z2U7TIXUWAAYNBMOILLY4OUCJG56HERDAF72B2JQAUA, seg_pos={0 0}, upload=439948]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 26 12:47:00.252251072 (501h18m0.123334117s) [noise=true , address=45.91.116.4:4937 , piece_id=KWHXM65G5Z2U7TIXUWAAYNBMOILLY4OUCJG56HERDAF72B2JQAUA, seg_pos={0 0}, upload=439948]: Long tail cancellation due to optimal threshold Dec 26 12:47:00.252271341 (501h18m0.123354376s) [seg_pos={0 0}, upload=439948]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:47:00.252348187 (501h18m0.123431232s) [upload=439948]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:47:00.252353377 (501h18m0.123436422s) [upload=439948]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:47:00.533600250 (501h18m0.404683315s) [upload=439948]: Done uploading segments... Dec 26 12:47:00.533628233 (501h18m0.404711278s) [upload=439948]: Upload finished. err: <nil> Duration: 30.00327501s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............................X....................X.XXX...] duration: 48.173190959s trace: https://jaeger.dev.storj.io/trace/399fe67f25bace88 FAIL (2025-12-26 12:47:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, Piece ID: 5BGLHPSJBADM4I66EOGEJYRLZFFPAHJ53MHVFDRSX24WUYY7YI2A) context deadline exceeded; piecestore: (Node ID: 12ZH3hS6xJp29U2RWAN9p8kxNdkSUQr1E58HzcFei48fUbNzFvW, Piece ID: FA6GAMWZ5EU23JWHVCFUY5ZJW6465F6ATA6RG4PH2H36URJVT65A) context deadline exceeded; piecestore: (Node ID: 1ftkFznfF2NSYrtgqUm3hgcyfwaZtWQfh2nDBEawxPh4SecemT, Piece ID: RSKD22V3Q23WFN4OVBYBLTSCOZJ2PXTVUSF4BIVIRSI6D2NC5YEA) context deadline exceeded ======LOGS====== EUHA, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.664140644 (501h14m37.535223679s) [noise=true , address=77.90.31.18:4822 , piece_id=2AL5F3CR5TACQXAHM7HGTTIHV6EUVF7P4NFVXO6V7ORAKBQM6QGA, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.664597445 (501h14m37.53568048s) [noise=true , address=145.79.227.34:4931 , piece_id=NMROTFKKS7GAMJZCZYLOIFDIUG6VP4D5W6HJ6B7DYVORMIQW5U6Q, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.665124648 (501h14m37.536207684s) [noise=true , address=77.90.31.21:4905 , piece_id=JV5W42QWWKR7ZZTRCEM2OVUIPWXFPTVMCDJ7NMF4YCPCJTB5WS5Q, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.665665368 (501h14m37.536748413s) [noise=true , address=145.79.227.44:4203 , piece_id=2KUGM74WCNULTSMYND6F32TXTWKPAP7UNNVRH3277TD2XUX3CSYA, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.666319684 (501h14m37.537402729s) [noise=true , address=77.90.31.13:4637 , piece_id=3RRZLEC4RZVM2UBMMIIJTQGQA5LHTQUFUUHKZFI57IU4MWC6ZJMQ, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.666533631 (501h14m37.537616676s) [noise=true , address=145.79.227.29:4965 , piece_id=D7YPJAELU2EGK72LINU773ZMFPMXRCH3YTDTAI3VSXDZXH5JCFFQ, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.667061025 (501h14m37.53814406s) [noise=true , address=145.79.227.27:4707 , piece_id=V3N5RRIZWZX72O4AZM2EQG34RU4FNWGBXO7JJSMYRQIBVI4UQ4RA, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.667575655 (501h14m37.53865869s) [noise=true , address=77.90.31.14:4686 , piece_id=5BGLHPSJBADM4I66EOGEJYRLZFFPAHJ53MHVFDRSX24WUYY7YI2A, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.668173774 (501h14m37.539256819s) [noise=true , address=77.90.31.12:4601 , piece_id=44WEMBXHWAGSGPPK5UQ2D2BL7OIAXYKNX2VV4Z26KZ6RIYQKSTFQ, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.668191278 (501h14m37.539274313s) [noise=true , address=45.91.116.14:4220 , piece_id=FA6GAMWZ5EU23JWHVCFUY5ZJW6465F6ATA6RG4PH2H36URJVT65A, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.668671292 (501h14m37.539754327s) [noise=true , address=145.79.227.40:4080 , piece_id=UQSPVT5RFKOHTN5TGLZWL5RYFR5TTR4FCUHNFZFCEDDZCROOBQVQ, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.669122792 (501h14m37.540205827s) [noise=true , address=77.90.31.17:4800 , piece_id=K72DLMFURND3TDQF3LRBH2D3BKGQBK73VKUGCHBWWENQUA3PB2OA, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.669621822 (501h14m37.540704867s) [noise=true , address=77.90.31.16:4764 , piece_id=RSKD22V3Q23WFN4OVBYBLTSCOZJ2PXTVUSF4BIVIRSI6D2NC5YEA, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.671130053 (501h14m37.542213088s) [noise=true , address=145.79.227.30:4767 , piece_id=64OMR2SYEU3TQYGEIBTLE3UAMWEA2C2R3XNWTCX2FKF3STLWD2YA, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.671287944 (501h14m37.542370979s) [noise=true , address=64.49.13.169:4706 , piece_id=LWAI7OPYOOSEJTWJU3WAJ4RRYOLBIHGGZUF7DHQR23ZKSGCDWQFA, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:39.141878555 (501h14m39.012961591s) [upload=439902]: Next returned nil segment Dec 26 12:43:39.141893735 (501h14m39.01297678s) [upload=439902]: Waiting for error group managing segments... Dec 26 12:43:39.537095507 (501h14m39.408178553s) [noise=true , address=145.79.227.24:4584 , piece_id=D5QC26FDHQ5TQBGFFWJUOOWUGIZSAYJS5KOOJFTAD6S7FW4YTFGA, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:39.544128439 (501h14m39.415211484s) [noise=true , address=145.79.227.37:4570 , piece_id=E6XQYXQ4JXMIJWF6J4DCJ4P6565N7JBWTLQ4GRFO6FYEMYPVATLQ, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:39.562038233 (501h14m39.433121278s) [noise=true , address=145.79.227.29:4965 , piece_id=D7YPJAELU2EGK72LINU773ZMFPMXRCH3YTDTAI3VSXDZXH5JCFFQ, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:39.577117587 (501h14m39.448200622s) [noise=true , address=145.79.227.27:4707 , piece_id=V3N5RRIZWZX72O4AZM2EQG34RU4FNWGBXO7JJSMYRQIBVI4UQ4RA, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:39.577340131 (501h14m39.448423166s) [noise=true , address=145.79.227.34:4931 , piece_id=NMROTFKKS7GAMJZCZYLOIFDIUG6VP4D5W6HJ6B7DYVORMIQW5U6Q, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:39.705570193 (501h14m39.576653238s) [noise=true , address=77.90.31.14:4686 , piece_id=5BGLHPSJBADM4I66EOGEJYRLZFFPAHJ53MHVFDRSX24WUYY7YI2A, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:39.772478088 (501h14m39.643561133s) [noise=true , address=45.91.116.14:4220 , piece_id=FA6GAMWZ5EU23JWHVCFUY5ZJW6465F6ATA6RG4PH2H36URJVT65A, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:39.831903704 (501h14m39.702986749s) [noise=true , address=77.90.31.13:4637 , piece_id=3RRZLEC4RZVM2UBMMIIJTQGQA5LHTQUFUUHKZFI57IU4MWC6ZJMQ, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:39.832432251 (501h14m39.703515286s) [noise=true , address=77.90.31.18:4822 , piece_id=2AL5F3CR5TACQXAHM7HGTTIHV6EUVF7P4NFVXO6V7ORAKBQM6QGA, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:39.844820801 (501h14m39.715903846s) [noise=true , address=77.90.31.16:4764 , piece_id=RSKD22V3Q23WFN4OVBYBLTSCOZJ2PXTVUSF4BIVIRSI6D2NC5YEA, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:39.854406263 (501h14m39.725489298s) [noise=true , address=77.90.31.15:4726 , piece_id=I47IGY4BJHX42ODE4P3CZVWNA5EBT7XKIBAUYUOAKXOWBCCUEUHA, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:40.410085534 (501h14m40.281168579s) [noise=true , address=145.79.227.44:4203 , piece_id=2KUGM74WCNULTSMYND6F32TXTWKPAP7UNNVRH3277TD2XUX3CSYA, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:40.488659412 (501h14m40.359742447s) [noise=true , address=145.79.227.30:4767 , piece_id=64OMR2SYEU3TQYGEIBTLE3UAMWEA2C2R3XNWTCX2FKF3STLWD2YA, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:40.684298553 (501h14m40.555381598s) [noise=true , address=77.90.31.17:4800 , piece_id=K72DLMFURND3TDQF3LRBH2D3BKGQBK73VKUGCHBWWENQUA3PB2OA, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:40.684320956 (501h14m40.555404001s) [seg_pos={0 0}, upload=439902]: Segment reached optimal threshold of 14 pieces. Dec 26 12:43:40.684595939 (501h14m40.555678994s) [noise=true , address=77.90.31.21:4905 , piece_id=JV5W42QWWKR7ZZTRCEM2OVUIPWXFPTVMCDJ7NMF4YCPCJTB5WS5Q, seg_pos={0 0}, upload=439902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK): protocol: expected piece hash; context canceled; EOF Dec 26 12:43:40.684606179 (501h14m40.555689214s) [noise=true , address=77.90.31.21:4905 , piece_id=JV5W42QWWKR7ZZTRCEM2OVUIPWXFPTVMCDJ7NMF4YCPCJTB5WS5Q, seg_pos={0 0}, upload=439902]: Long tail cancellation due to optimal threshold Dec 26 12:43:40.684627660 (501h14m40.555710715s) [noise=true , address=64.49.13.169:4706 , piece_id=LWAI7OPYOOSEJTWJU3WAJ4RRYOLBIHGGZUF7DHQR23ZKSGCDWQFA, seg_pos={0 0}, upload=439902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:41264->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:41264->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41264->64.49.13.169:4706: use of closed network connection Dec 26 12:43:40.684637739 (501h14m40.555720784s) [noise=true , address=64.49.13.169:4706 , piece_id=LWAI7OPYOOSEJTWJU3WAJ4RRYOLBIHGGZUF7DHQR23ZKSGCDWQFA, seg_pos={0 0}, upload=439902]: Long tail cancellation due to optimal threshold Dec 26 12:43:40.684840836 (501h14m40.555923891s) [noise=true , address=77.90.31.12:4601 , piece_id=44WEMBXHWAGSGPPK5UQ2D2BL7OIAXYKNX2VV4Z26KZ6RIYQKSTFQ, seg_pos={0 0}, upload=439902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pQPpDBX1nYAGc961a72T8NyFRc8yLMKfvYBqwuNBYQetzQZL2): protocol: read tcp 188.166.225.170:56438->77.90.31.12:4601: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 12:43:40.684849853 (501h14m40.555932898s) [noise=true , address=77.90.31.12:4601 , piece_id=44WEMBXHWAGSGPPK5UQ2D2BL7OIAXYKNX2VV4Z26KZ6RIYQKSTFQ, seg_pos={0 0}, upload=439902]: Long tail cancellation due to optimal threshold Dec 26 12:43:40.684993706 (501h14m40.556076741s) [noise=true , address=145.79.227.40:4080 , piece_id=UQSPVT5RFKOHTN5TGLZWL5RYFR5TTR4FCUHNFZFCEDDZCROOBQVQ, seg_pos={0 0}, upload=439902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WXMdaUQnFnXfayz6CkmsK1wuLhWSpXjQQt2kZyVTVnv3qL44C): protocol: expected piece hash; context canceled; EOF Dec 26 12:43:40.685002634 (501h14m40.556085679s) [noise=true , address=145.79.227.40:4080 , piece_id=UQSPVT5RFKOHTN5TGLZWL5RYFR5TTR4FCUHNFZFCEDDZCROOBQVQ, seg_pos={0 0}, upload=439902]: Long tail cancellation due to optimal threshold Dec 26 12:43:40.685024215 (501h14m40.55610725s) [seg_pos={0 0}, upload=439902]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 12:43:40.685126900 (501h14m40.556209945s) [upload=439902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:43:40.685134925 (501h14m40.55621796s) [upload=439902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:43:40.978747536 (501h14m40.849830581s) [upload=439902]: Done uploading segments... Dec 26 12:43:40.978770249 (501h14m40.849853294s) [upload=439902]: Upload finished. err: <nil> Duration: 48.173214575s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [....X..X......X.......XXX...X.....X.......X.XX......XXX..XXX] duration: 5.001009508s trace: https://jaeger.dev.storj.io/trace/1b4ffc32211f0f20 FAIL (2025-12-26 12:47:05 UTC) orbiter warning: context deadline exceeded Duration: 5.001036008s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [................................X....................X.XXX..] duration: 52.271192248s trace: https://jaeger.dev.storj.io/trace/399fe67f25bace88 FAIL (2025-12-26 12:46:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, Piece ID: 5BGLHPSJBADM4I66EOGEJYRLZFFPAHJ53MHVFDRSX24WUYY7YI2A) context deadline exceeded; piecestore: (Node ID: 12ZH3hS6xJp29U2RWAN9p8kxNdkSUQr1E58HzcFei48fUbNzFvW, Piece ID: FA6GAMWZ5EU23JWHVCFUY5ZJW6465F6ATA6RG4PH2H36URJVT65A) context deadline exceeded; piecestore: (Node ID: 1ftkFznfF2NSYrtgqUm3hgcyfwaZtWQfh2nDBEawxPh4SecemT, Piece ID: RSKD22V3Q23WFN4OVBYBLTSCOZJ2PXTVUSF4BIVIRSI6D2NC5YEA) context deadline exceeded ======LOGS====== EUHA, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.664140644 (501h14m37.535223679s) [noise=true , address=77.90.31.18:4822 , piece_id=2AL5F3CR5TACQXAHM7HGTTIHV6EUVF7P4NFVXO6V7ORAKBQM6QGA, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.664597445 (501h14m37.53568048s) [noise=true , address=145.79.227.34:4931 , piece_id=NMROTFKKS7GAMJZCZYLOIFDIUG6VP4D5W6HJ6B7DYVORMIQW5U6Q, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.665124648 (501h14m37.536207684s) [noise=true , address=77.90.31.21:4905 , piece_id=JV5W42QWWKR7ZZTRCEM2OVUIPWXFPTVMCDJ7NMF4YCPCJTB5WS5Q, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.665665368 (501h14m37.536748413s) [noise=true , address=145.79.227.44:4203 , piece_id=2KUGM74WCNULTSMYND6F32TXTWKPAP7UNNVRH3277TD2XUX3CSYA, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.666319684 (501h14m37.537402729s) [noise=true , address=77.90.31.13:4637 , piece_id=3RRZLEC4RZVM2UBMMIIJTQGQA5LHTQUFUUHKZFI57IU4MWC6ZJMQ, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.666533631 (501h14m37.537616676s) [noise=true , address=145.79.227.29:4965 , piece_id=D7YPJAELU2EGK72LINU773ZMFPMXRCH3YTDTAI3VSXDZXH5JCFFQ, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.667061025 (501h14m37.53814406s) [noise=true , address=145.79.227.27:4707 , piece_id=V3N5RRIZWZX72O4AZM2EQG34RU4FNWGBXO7JJSMYRQIBVI4UQ4RA, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.667575655 (501h14m37.53865869s) [noise=true , address=77.90.31.14:4686 , piece_id=5BGLHPSJBADM4I66EOGEJYRLZFFPAHJ53MHVFDRSX24WUYY7YI2A, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.668173774 (501h14m37.539256819s) [noise=true , address=77.90.31.12:4601 , piece_id=44WEMBXHWAGSGPPK5UQ2D2BL7OIAXYKNX2VV4Z26KZ6RIYQKSTFQ, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.668191278 (501h14m37.539274313s) [noise=true , address=45.91.116.14:4220 , piece_id=FA6GAMWZ5EU23JWHVCFUY5ZJW6465F6ATA6RG4PH2H36URJVT65A, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.668671292 (501h14m37.539754327s) [noise=true , address=145.79.227.40:4080 , piece_id=UQSPVT5RFKOHTN5TGLZWL5RYFR5TTR4FCUHNFZFCEDDZCROOBQVQ, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.669122792 (501h14m37.540205827s) [noise=true , address=77.90.31.17:4800 , piece_id=K72DLMFURND3TDQF3LRBH2D3BKGQBK73VKUGCHBWWENQUA3PB2OA, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.669621822 (501h14m37.540704867s) [noise=true , address=77.90.31.16:4764 , piece_id=RSKD22V3Q23WFN4OVBYBLTSCOZJ2PXTVUSF4BIVIRSI6D2NC5YEA, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.671130053 (501h14m37.542213088s) [noise=true , address=145.79.227.30:4767 , piece_id=64OMR2SYEU3TQYGEIBTLE3UAMWEA2C2R3XNWTCX2FKF3STLWD2YA, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.671287944 (501h14m37.542370979s) [noise=true , address=64.49.13.169:4706 , piece_id=LWAI7OPYOOSEJTWJU3WAJ4RRYOLBIHGGZUF7DHQR23ZKSGCDWQFA, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:39.141878555 (501h14m39.012961591s) [upload=439902]: Next returned nil segment Dec 26 12:43:39.141893735 (501h14m39.01297678s) [upload=439902]: Waiting for error group managing segments... Dec 26 12:43:39.537095507 (501h14m39.408178553s) [noise=true , address=145.79.227.24:4584 , piece_id=D5QC26FDHQ5TQBGFFWJUOOWUGIZSAYJS5KOOJFTAD6S7FW4YTFGA, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:39.544128439 (501h14m39.415211484s) [noise=true , address=145.79.227.37:4570 , piece_id=E6XQYXQ4JXMIJWF6J4DCJ4P6565N7JBWTLQ4GRFO6FYEMYPVATLQ, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:39.562038233 (501h14m39.433121278s) [noise=true , address=145.79.227.29:4965 , piece_id=D7YPJAELU2EGK72LINU773ZMFPMXRCH3YTDTAI3VSXDZXH5JCFFQ, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:39.577117587 (501h14m39.448200622s) [noise=true , address=145.79.227.27:4707 , piece_id=V3N5RRIZWZX72O4AZM2EQG34RU4FNWGBXO7JJSMYRQIBVI4UQ4RA, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:39.577340131 (501h14m39.448423166s) [noise=true , address=145.79.227.34:4931 , piece_id=NMROTFKKS7GAMJZCZYLOIFDIUG6VP4D5W6HJ6B7DYVORMIQW5U6Q, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:39.705570193 (501h14m39.576653238s) [noise=true , address=77.90.31.14:4686 , piece_id=5BGLHPSJBADM4I66EOGEJYRLZFFPAHJ53MHVFDRSX24WUYY7YI2A, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:39.772478088 (501h14m39.643561133s) [noise=true , address=45.91.116.14:4220 , piece_id=FA6GAMWZ5EU23JWHVCFUY5ZJW6465F6ATA6RG4PH2H36URJVT65A, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:39.831903704 (501h14m39.702986749s) [noise=true , address=77.90.31.13:4637 , piece_id=3RRZLEC4RZVM2UBMMIIJTQGQA5LHTQUFUUHKZFI57IU4MWC6ZJMQ, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:39.832432251 (501h14m39.703515286s) [noise=true , address=77.90.31.18:4822 , piece_id=2AL5F3CR5TACQXAHM7HGTTIHV6EUVF7P4NFVXO6V7ORAKBQM6QGA, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:39.844820801 (501h14m39.715903846s) [noise=true , address=77.90.31.16:4764 , piece_id=RSKD22V3Q23WFN4OVBYBLTSCOZJ2PXTVUSF4BIVIRSI6D2NC5YEA, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:39.854406263 (501h14m39.725489298s) [noise=true , address=77.90.31.15:4726 , piece_id=I47IGY4BJHX42ODE4P3CZVWNA5EBT7XKIBAUYUOAKXOWBCCUEUHA, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:40.410085534 (501h14m40.281168579s) [noise=true , address=145.79.227.44:4203 , piece_id=2KUGM74WCNULTSMYND6F32TXTWKPAP7UNNVRH3277TD2XUX3CSYA, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:40.488659412 (501h14m40.359742447s) [noise=true , address=145.79.227.30:4767 , piece_id=64OMR2SYEU3TQYGEIBTLE3UAMWEA2C2R3XNWTCX2FKF3STLWD2YA, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:40.684298553 (501h14m40.555381598s) [noise=true , address=77.90.31.17:4800 , piece_id=K72DLMFURND3TDQF3LRBH2D3BKGQBK73VKUGCHBWWENQUA3PB2OA, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:40.684320956 (501h14m40.555404001s) [seg_pos={0 0}, upload=439902]: Segment reached optimal threshold of 14 pieces. Dec 26 12:43:40.684595939 (501h14m40.555678994s) [noise=true , address=77.90.31.21:4905 , piece_id=JV5W42QWWKR7ZZTRCEM2OVUIPWXFPTVMCDJ7NMF4YCPCJTB5WS5Q, seg_pos={0 0}, upload=439902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK): protocol: expected piece hash; context canceled; EOF Dec 26 12:43:40.684606179 (501h14m40.555689214s) [noise=true , address=77.90.31.21:4905 , piece_id=JV5W42QWWKR7ZZTRCEM2OVUIPWXFPTVMCDJ7NMF4YCPCJTB5WS5Q, seg_pos={0 0}, upload=439902]: Long tail cancellation due to optimal threshold Dec 26 12:43:40.684627660 (501h14m40.555710715s) [noise=true , address=64.49.13.169:4706 , piece_id=LWAI7OPYOOSEJTWJU3WAJ4RRYOLBIHGGZUF7DHQR23ZKSGCDWQFA, seg_pos={0 0}, upload=439902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:41264->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:41264->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41264->64.49.13.169:4706: use of closed network connection Dec 26 12:43:40.684637739 (501h14m40.555720784s) [noise=true , address=64.49.13.169:4706 , piece_id=LWAI7OPYOOSEJTWJU3WAJ4RRYOLBIHGGZUF7DHQR23ZKSGCDWQFA, seg_pos={0 0}, upload=439902]: Long tail cancellation due to optimal threshold Dec 26 12:43:40.684840836 (501h14m40.555923891s) [noise=true , address=77.90.31.12:4601 , piece_id=44WEMBXHWAGSGPPK5UQ2D2BL7OIAXYKNX2VV4Z26KZ6RIYQKSTFQ, seg_pos={0 0}, upload=439902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pQPpDBX1nYAGc961a72T8NyFRc8yLMKfvYBqwuNBYQetzQZL2): protocol: read tcp 188.166.225.170:56438->77.90.31.12:4601: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 12:43:40.684849853 (501h14m40.555932898s) [noise=true , address=77.90.31.12:4601 , piece_id=44WEMBXHWAGSGPPK5UQ2D2BL7OIAXYKNX2VV4Z26KZ6RIYQKSTFQ, seg_pos={0 0}, upload=439902]: Long tail cancellation due to optimal threshold Dec 26 12:43:40.684993706 (501h14m40.556076741s) [noise=true , address=145.79.227.40:4080 , piece_id=UQSPVT5RFKOHTN5TGLZWL5RYFR5TTR4FCUHNFZFCEDDZCROOBQVQ, seg_pos={0 0}, upload=439902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WXMdaUQnFnXfayz6CkmsK1wuLhWSpXjQQt2kZyVTVnv3qL44C): protocol: expected piece hash; context canceled; EOF Dec 26 12:43:40.685002634 (501h14m40.556085679s) [noise=true , address=145.79.227.40:4080 , piece_id=UQSPVT5RFKOHTN5TGLZWL5RYFR5TTR4FCUHNFZFCEDDZCROOBQVQ, seg_pos={0 0}, upload=439902]: Long tail cancellation due to optimal threshold Dec 26 12:43:40.685024215 (501h14m40.55610725s) [seg_pos={0 0}, upload=439902]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 12:43:40.685126900 (501h14m40.556209945s) [upload=439902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:43:40.685134925 (501h14m40.55621796s) [upload=439902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:43:40.978747536 (501h14m40.849830581s) [upload=439902]: Done uploading segments... Dec 26 12:43:40.978770249 (501h14m40.849853294s) [upload=439902]: Upload finished. err: <nil> Duration: 52.271211955s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..X.X....X..X...X..XX.....XXXX.....XXX....XX....X..X......X.] duration: 6.2603333s trace: https://jaeger.dev.storj.io/trace/7cb46c2c5d33667b FAIL (2025-12-26 12:46:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: C24I5YETHX7QXOGEGTLFPIT7WJ4I5KCTDRC4GLNVJ5AI2HMSOZWQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: NC57EI6ZMEQTBVDGZCCAVSCYMZAYUGPRNYDCAL67VIC4JPQOZZ6Q) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: ITEU2CA4H3MXUZMI72O4BSWTN47CBX6UK3IV2P3YJUNPPEXU7SLA) context deadline exceeded ======LOGS====== nts... Dec 26 12:45:38.491303757 (501h16m38.362386792s) [upload=439933]: Got next segment. Inline: false Dec 26 12:45:38.491334827 (501h16m38.362417862s) [upload=439933]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:45:38.491339235 (501h16m38.36242228s) [upload=439933]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:45:38.735917094 (501h16m38.607000139s) [seg_pos={0 0}, upload=439933]: Begin upload segment... Dec 26 12:45:38.735979323 (501h16m38.607062398s) [seg_pos={0 0}, upload=439933]: Done begin upload segment. Dec 26 12:45:38.736028536 (501h16m38.607111571s) [noise=true , address=145.79.227.4:4185 , piece_id=RPGG36UJMBP3W6VRPXBXHOMCLD53SC2HQFYYCJKG5DZK3DL37RLQ, seg_pos={0 0}, upload=439933]: Uploading piece... Dec 26 12:45:38.736065226 (501h16m38.607148271s) [noise=true , address=45.91.116.12:4228 , piece_id=XVBGLE3BIPGLXH2F3FXAHXSAFSRW5R4AWNPNINIIAHXDWJ4Z4MYA, seg_pos={0 0}, upload=439933]: Uploading piece... Dec 26 12:45:38.736769187 (501h16m38.607852232s) [noise=true , address=77.90.31.14:4663 , piece_id=C24I5YETHX7QXOGEGTLFPIT7WJ4I5KCTDRC4GLNVJ5AI2HMSOZWQ, seg_pos={0 0}, upload=439933]: Uploading piece... Dec 26 12:45:38.736942377 (501h16m38.608025422s) [noise=true , address=145.79.227.19:4078 , piece_id=CAFFQUH6XPX7LDFO5KXVMJWQQBGMEODP56XTD2ZC4DHJTRDMXM4Q, seg_pos={0 0}, upload=439933]: Uploading piece... Dec 26 12:45:38.737283576 (501h16m38.608366621s) [noise=true , address=77.90.31.10:4519 , piece_id=2AZN5NPX7JMYCMRGJSEM4B3NOKF4Y4KAJCNU7NMFPIHEXWS5JBBQ, seg_pos={0 0}, upload=439933]: Uploading piece... Dec 26 12:45:38.737322100 (501h16m38.608405155s) [noise=true , address=145.79.227.32:4834 , piece_id=6EAMXXCFN7SLLPHYX4TXF45M4J6TRLLRGEFZVNVSEHXDTXZC42LA, seg_pos={0 0}, upload=439933]: Uploading piece... Dec 26 12:45:38.737596091 (501h16m38.608679136s) [noise=true , address=45.91.116.9:4262 , piece_id=NC57EI6ZMEQTBVDGZCCAVSCYMZAYUGPRNYDCAL67VIC4JPQOZZ6Q, seg_pos={0 0}, upload=439933]: Uploading piece... Dec 26 12:45:38.737977518 (501h16m38.609060573s) [noise=true , address=145.79.227.40:4051 , piece_id=YOFKBE7NN35M4SKC5JCLREPXPD2UZ4H5O5TX4PG7GTGEMQEPOUZA, seg_pos={0 0}, upload=439933]: Uploading piece... Dec 26 12:45:38.737988659 (501h16m38.609071715s) [noise=true , address=145.79.227.46:4601 , piece_id=5C3PSC5LMSRTOXNUG5AWKXJIV6BM7OMRCZHF3Q3T43QBHGHRL7IA, seg_pos={0 0}, upload=439933]: Uploading piece... Dec 26 12:45:38.738076907 (501h16m38.609159952s) [noise=true , address=145.79.227.52:4445 , piece_id=M35PVAXJSCZ2XDMBDOET6PYLL56N2ZOTCNEVLJSIZBWZ6COMWLTA, seg_pos={0 0}, upload=439933]: Uploading piece... Dec 26 12:45:38.738409310 (501h16m38.609492345s) [noise=true , address=45.91.116.14:4190 , piece_id=PHUFBC3OWAZOJMRGSWW56ZG4CFTTBUQP22TZRTTQYL2F7VJADWUA, seg_pos={0 0}, upload=439933]: Uploading piece... Dec 26 12:45:38.738739398 (501h16m38.609822443s) [noise=true , address=145.79.227.16:4910 , piece_id=K5C3AEKGONR6NM2TC5B4FRHP4CCA2IT2SY6X3C3SNXRXYKKR3NKQ, seg_pos={0 0}, upload=439933]: Uploading piece... Dec 26 12:45:38.738905625 (501h16m38.60998866s) [noise=true , address=45.91.116.5:4299 , piece_id=ITEU2CA4H3MXUZMI72O4BSWTN47CBX6UK3IV2P3YJUNPPEXU7SLA, seg_pos={0 0}, upload=439933]: Uploading piece... Dec 26 12:45:38.739305246 (501h16m38.610388291s) [noise=true , address=145.79.227.41:4086 , piece_id=VSPM6EIO5IKJB23IB5JYYV7CUJDZYYCBAFP3MAOBFWNA5UQNJWXQ, seg_pos={0 0}, upload=439933]: Uploading piece... Dec 26 12:45:38.739327568 (501h16m38.610410614s) [noise=true , address=145.79.227.3:4149 , piece_id=K6KQEGTIPZN4WI3SUPTFPORE4EKZD2QP5NRBLX7WZS4QILBUUJ2A, seg_pos={0 0}, upload=439933]: Uploading piece... Dec 26 12:45:38.739969972 (501h16m38.611053017s) [noise=true , address=45.91.116.13:4153 , piece_id=L7EC4Z62EWI6C4BTN7G7RZLRNUAHDVY6FLPVXB6MCIXMXDVRJDDA, seg_pos={0 0}, upload=439933]: Uploading piece... Dec 26 12:45:40.456247174 (501h16m40.327330209s) [upload=439933]: Next returned nil segment Dec 26 12:45:40.456260409 (501h16m40.327343454s) [upload=439933]: Waiting for error group managing segments... Dec 26 12:45:40.744014624 (501h16m40.615097669s) [noise=true , address=145.79.227.19:4078 , piece_id=CAFFQUH6XPX7LDFO5KXVMJWQQBGMEODP56XTD2ZC4DHJTRDMXM4Q, seg_pos={0 0}, upload=439933]: Done uploading piece. err: <nil> Dec 26 12:45:40.746305115 (501h16m40.61738815s) [noise=true , address=145.79.227.16:4910 , piece_id=K5C3AEKGONR6NM2TC5B4FRHP4CCA2IT2SY6X3C3SNXRXYKKR3NKQ, seg_pos={0 0}, upload=439933]: Done uploading piece. err: <nil> Dec 26 12:45:40.749079097 (501h16m40.620162142s) [noise=true , address=145.79.227.52:4445 , piece_id=M35PVAXJSCZ2XDMBDOET6PYLL56N2ZOTCNEVLJSIZBWZ6COMWLTA, seg_pos={0 0}, upload=439933]: Done uploading piece. err: <nil> Dec 26 12:45:40.755744419 (501h16m40.626827454s) [noise=true , address=145.79.227.3:4149 , piece_id=K6KQEGTIPZN4WI3SUPTFPORE4EKZD2QP5NRBLX7WZS4QILBUUJ2A, seg_pos={0 0}, upload=439933]: Done uploading piece. err: <nil> Dec 26 12:45:40.756452297 (501h16m40.627535332s) [noise=true , address=145.79.227.46:4601 , piece_id=5C3PSC5LMSRTOXNUG5AWKXJIV6BM7OMRCZHF3Q3T43QBHGHRL7IA, seg_pos={0 0}, upload=439933]: Done uploading piece. err: <nil> Dec 26 12:45:40.762810614 (501h16m40.633893649s) [noise=true , address=145.79.227.41:4086 , piece_id=VSPM6EIO5IKJB23IB5JYYV7CUJDZYYCBAFP3MAOBFWNA5UQNJWXQ, seg_pos={0 0}, upload=439933]: Done uploading piece. err: <nil> Dec 26 12:45:40.764816053 (501h16m40.635899089s) [noise=true , address=145.79.227.4:4185 , piece_id=RPGG36UJMBP3W6VRPXBXHOMCLD53SC2HQFYYCJKG5DZK3DL37RLQ, seg_pos={0 0}, upload=439933]: Done uploading piece. err: <nil> Dec 26 12:45:40.767299081 (501h16m40.638382126s) [noise=true , address=145.79.227.40:4051 , piece_id=YOFKBE7NN35M4SKC5JCLREPXPD2UZ4H5O5TX4PG7GTGEMQEPOUZA, seg_pos={0 0}, upload=439933]: Done uploading piece. err: <nil> Dec 26 12:45:40.769117454 (501h16m40.640200489s) [noise=true , address=145.79.227.32:4834 , piece_id=6EAMXXCFN7SLLPHYX4TXF45M4J6TRLLRGEFZVNVSEHXDTXZC42LA, seg_pos={0 0}, upload=439933]: Done uploading piece. err: <nil> Dec 26 12:45:41.010937561 (501h16m40.882020596s) [noise=true , address=45.91.116.9:4262 , piece_id=NC57EI6ZMEQTBVDGZCCAVSCYMZAYUGPRNYDCAL67VIC4JPQOZZ6Q, seg_pos={0 0}, upload=439933]: Done uploading piece. err: <nil> Dec 26 12:45:41.059442423 (501h16m40.930525458s) [noise=true , address=77.90.31.14:4663 , piece_id=C24I5YETHX7QXOGEGTLFPIT7WJ4I5KCTDRC4GLNVJ5AI2HMSOZWQ, seg_pos={0 0}, upload=439933]: Done uploading piece. err: <nil> Dec 26 12:45:41.301071478 (501h16m41.172154533s) [noise=true , address=45.91.116.5:4299 , piece_id=ITEU2CA4H3MXUZMI72O4BSWTN47CBX6UK3IV2P3YJUNPPEXU7SLA, seg_pos={0 0}, upload=439933]: Done uploading piece. err: <nil> Dec 26 12:45:41.397894325 (501h16m41.26897736s) [noise=true , address=77.90.31.10:4519 , piece_id=2AZN5NPX7JMYCMRGJSEM4B3NOKF4Y4KAJCNU7NMFPIHEXWS5JBBQ, seg_pos={0 0}, upload=439933]: Done uploading piece. err: <nil> Dec 26 12:45:41.397914042 (501h16m41.268997078s) [seg_pos={0 0}, upload=439933]: Segment reached optimal threshold of 13 pieces. Dec 26 12:45:41.398126918 (501h16m41.269209953s) [noise=true , address=45.91.116.12:4228 , piece_id=XVBGLE3BIPGLXH2F3FXAHXSAFSRW5R4AWNPNINIIAHXDWJ4Z4MYA, seg_pos={0 0}, upload=439933]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:59016->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:59016->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59016->45.91.116.12:4228: use of closed network connection Dec 26 12:45:41.398134352 (501h16m41.269217387s) [noise=true , address=45.91.116.12:4228 , piece_id=XVBGLE3BIPGLXH2F3FXAHXSAFSRW5R4AWNPNINIIAHXDWJ4Z4MYA, seg_pos={0 0}, upload=439933]: Long tail cancellation due to optimal threshold Dec 26 12:45:41.398245023 (501h16m41.269328068s) [noise=true , address=45.91.116.13:4153 , piece_id=L7EC4Z62EWI6C4BTN7G7RZLRNUAHDVY6FLPVXB6MCIXMXDVRJDDA, seg_pos={0 0}, upload=439933]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:35936->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:35936->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35936->45.91.116.13:4153: use of closed network connection Dec 26 12:45:41.398274619 (501h16m41.269357654s) [noise=true , address=45.91.116.13:4153 , piece_id=L7EC4Z62EWI6C4BTN7G7RZLRNUAHDVY6FLPVXB6MCIXMXDVRJDDA, seg_pos={0 0}, upload=439933]: Long tail cancellation due to optimal threshold Dec 26 12:45:41.398284669 (501h16m41.269367714s) [noise=true , address=45.91.116.14:4190 , piece_id=PHUFBC3OWAZOJMRGSWW56ZG4CFTTBUQP22TZRTTQYL2F7VJADWUA, seg_pos={0 0}, upload=439933]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 26 12:45:41.398291692 (501h16m41.269374737s) [noise=true , address=45.91.116.14:4190 , piece_id=PHUFBC3OWAZOJMRGSWW56ZG4CFTTBUQP22TZRTTQYL2F7VJADWUA, seg_pos={0 0}, upload=439933]: Long tail cancellation due to optimal threshold Dec 26 12:45:41.398371283 (501h16m41.269454328s) [seg_pos={0 0}, upload=439933]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:45:41.398544663 (501h16m41.269627698s) [upload=439933]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:45:41.398549813 (501h16m41.269632848s) [upload=439933]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:45:41.694499974 (501h16m41.565583019s) [upload=439933]: Done uploading segments... Dec 26 12:45:41.694523549 (501h16m41.565606584s) [upload=439933]: Upload finished. err: <nil> Duration: 6.26035984s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [...X.X....X..X...X..XX.....XXXX.....XXX....XX....X..X......X] duration: 30.004971598s trace: https://jaeger.dev.storj.io/trace/7cb46c2c5d33667b FAIL (2025-12-26 12:46:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: C24I5YETHX7QXOGEGTLFPIT7WJ4I5KCTDRC4GLNVJ5AI2HMSOZWQ) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: NC57EI6ZMEQTBVDGZCCAVSCYMZAYUGPRNYDCAL67VIC4JPQOZZ6Q) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: ITEU2CA4H3MXUZMI72O4BSWTN47CBX6UK3IV2P3YJUNPPEXU7SLA) context deadline exceeded ======LOGS====== nts... Dec 26 12:45:38.491303757 (501h16m38.362386792s) [upload=439933]: Got next segment. Inline: false Dec 26 12:45:38.491334827 (501h16m38.362417862s) [upload=439933]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:45:38.491339235 (501h16m38.36242228s) [upload=439933]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:45:38.735917094 (501h16m38.607000139s) [seg_pos={0 0}, upload=439933]: Begin upload segment... Dec 26 12:45:38.735979323 (501h16m38.607062398s) [seg_pos={0 0}, upload=439933]: Done begin upload segment. Dec 26 12:45:38.736028536 (501h16m38.607111571s) [noise=true , address=145.79.227.4:4185 , piece_id=RPGG36UJMBP3W6VRPXBXHOMCLD53SC2HQFYYCJKG5DZK3DL37RLQ, seg_pos={0 0}, upload=439933]: Uploading piece... Dec 26 12:45:38.736065226 (501h16m38.607148271s) [noise=true , address=45.91.116.12:4228 , piece_id=XVBGLE3BIPGLXH2F3FXAHXSAFSRW5R4AWNPNINIIAHXDWJ4Z4MYA, seg_pos={0 0}, upload=439933]: Uploading piece... Dec 26 12:45:38.736769187 (501h16m38.607852232s) [noise=true , address=77.90.31.14:4663 , piece_id=C24I5YETHX7QXOGEGTLFPIT7WJ4I5KCTDRC4GLNVJ5AI2HMSOZWQ, seg_pos={0 0}, upload=439933]: Uploading piece... Dec 26 12:45:38.736942377 (501h16m38.608025422s) [noise=true , address=145.79.227.19:4078 , piece_id=CAFFQUH6XPX7LDFO5KXVMJWQQBGMEODP56XTD2ZC4DHJTRDMXM4Q, seg_pos={0 0}, upload=439933]: Uploading piece... Dec 26 12:45:38.737283576 (501h16m38.608366621s) [noise=true , address=77.90.31.10:4519 , piece_id=2AZN5NPX7JMYCMRGJSEM4B3NOKF4Y4KAJCNU7NMFPIHEXWS5JBBQ, seg_pos={0 0}, upload=439933]: Uploading piece... Dec 26 12:45:38.737322100 (501h16m38.608405155s) [noise=true , address=145.79.227.32:4834 , piece_id=6EAMXXCFN7SLLPHYX4TXF45M4J6TRLLRGEFZVNVSEHXDTXZC42LA, seg_pos={0 0}, upload=439933]: Uploading piece... Dec 26 12:45:38.737596091 (501h16m38.608679136s) [noise=true , address=45.91.116.9:4262 , piece_id=NC57EI6ZMEQTBVDGZCCAVSCYMZAYUGPRNYDCAL67VIC4JPQOZZ6Q, seg_pos={0 0}, upload=439933]: Uploading piece... Dec 26 12:45:38.737977518 (501h16m38.609060573s) [noise=true , address=145.79.227.40:4051 , piece_id=YOFKBE7NN35M4SKC5JCLREPXPD2UZ4H5O5TX4PG7GTGEMQEPOUZA, seg_pos={0 0}, upload=439933]: Uploading piece... Dec 26 12:45:38.737988659 (501h16m38.609071715s) [noise=true , address=145.79.227.46:4601 , piece_id=5C3PSC5LMSRTOXNUG5AWKXJIV6BM7OMRCZHF3Q3T43QBHGHRL7IA, seg_pos={0 0}, upload=439933]: Uploading piece... Dec 26 12:45:38.738076907 (501h16m38.609159952s) [noise=true , address=145.79.227.52:4445 , piece_id=M35PVAXJSCZ2XDMBDOET6PYLL56N2ZOTCNEVLJSIZBWZ6COMWLTA, seg_pos={0 0}, upload=439933]: Uploading piece... Dec 26 12:45:38.738409310 (501h16m38.609492345s) [noise=true , address=45.91.116.14:4190 , piece_id=PHUFBC3OWAZOJMRGSWW56ZG4CFTTBUQP22TZRTTQYL2F7VJADWUA, seg_pos={0 0}, upload=439933]: Uploading piece... Dec 26 12:45:38.738739398 (501h16m38.609822443s) [noise=true , address=145.79.227.16:4910 , piece_id=K5C3AEKGONR6NM2TC5B4FRHP4CCA2IT2SY6X3C3SNXRXYKKR3NKQ, seg_pos={0 0}, upload=439933]: Uploading piece... Dec 26 12:45:38.738905625 (501h16m38.60998866s) [noise=true , address=45.91.116.5:4299 , piece_id=ITEU2CA4H3MXUZMI72O4BSWTN47CBX6UK3IV2P3YJUNPPEXU7SLA, seg_pos={0 0}, upload=439933]: Uploading piece... Dec 26 12:45:38.739305246 (501h16m38.610388291s) [noise=true , address=145.79.227.41:4086 , piece_id=VSPM6EIO5IKJB23IB5JYYV7CUJDZYYCBAFP3MAOBFWNA5UQNJWXQ, seg_pos={0 0}, upload=439933]: Uploading piece... Dec 26 12:45:38.739327568 (501h16m38.610410614s) [noise=true , address=145.79.227.3:4149 , piece_id=K6KQEGTIPZN4WI3SUPTFPORE4EKZD2QP5NRBLX7WZS4QILBUUJ2A, seg_pos={0 0}, upload=439933]: Uploading piece... Dec 26 12:45:38.739969972 (501h16m38.611053017s) [noise=true , address=45.91.116.13:4153 , piece_id=L7EC4Z62EWI6C4BTN7G7RZLRNUAHDVY6FLPVXB6MCIXMXDVRJDDA, seg_pos={0 0}, upload=439933]: Uploading piece... Dec 26 12:45:40.456247174 (501h16m40.327330209s) [upload=439933]: Next returned nil segment Dec 26 12:45:40.456260409 (501h16m40.327343454s) [upload=439933]: Waiting for error group managing segments... Dec 26 12:45:40.744014624 (501h16m40.615097669s) [noise=true , address=145.79.227.19:4078 , piece_id=CAFFQUH6XPX7LDFO5KXVMJWQQBGMEODP56XTD2ZC4DHJTRDMXM4Q, seg_pos={0 0}, upload=439933]: Done uploading piece. err: <nil> Dec 26 12:45:40.746305115 (501h16m40.61738815s) [noise=true , address=145.79.227.16:4910 , piece_id=K5C3AEKGONR6NM2TC5B4FRHP4CCA2IT2SY6X3C3SNXRXYKKR3NKQ, seg_pos={0 0}, upload=439933]: Done uploading piece. err: <nil> Dec 26 12:45:40.749079097 (501h16m40.620162142s) [noise=true , address=145.79.227.52:4445 , piece_id=M35PVAXJSCZ2XDMBDOET6PYLL56N2ZOTCNEVLJSIZBWZ6COMWLTA, seg_pos={0 0}, upload=439933]: Done uploading piece. err: <nil> Dec 26 12:45:40.755744419 (501h16m40.626827454s) [noise=true , address=145.79.227.3:4149 , piece_id=K6KQEGTIPZN4WI3SUPTFPORE4EKZD2QP5NRBLX7WZS4QILBUUJ2A, seg_pos={0 0}, upload=439933]: Done uploading piece. err: <nil> Dec 26 12:45:40.756452297 (501h16m40.627535332s) [noise=true , address=145.79.227.46:4601 , piece_id=5C3PSC5LMSRTOXNUG5AWKXJIV6BM7OMRCZHF3Q3T43QBHGHRL7IA, seg_pos={0 0}, upload=439933]: Done uploading piece. err: <nil> Dec 26 12:45:40.762810614 (501h16m40.633893649s) [noise=true , address=145.79.227.41:4086 , piece_id=VSPM6EIO5IKJB23IB5JYYV7CUJDZYYCBAFP3MAOBFWNA5UQNJWXQ, seg_pos={0 0}, upload=439933]: Done uploading piece. err: <nil> Dec 26 12:45:40.764816053 (501h16m40.635899089s) [noise=true , address=145.79.227.4:4185 , piece_id=RPGG36UJMBP3W6VRPXBXHOMCLD53SC2HQFYYCJKG5DZK3DL37RLQ, seg_pos={0 0}, upload=439933]: Done uploading piece. err: <nil> Dec 26 12:45:40.767299081 (501h16m40.638382126s) [noise=true , address=145.79.227.40:4051 , piece_id=YOFKBE7NN35M4SKC5JCLREPXPD2UZ4H5O5TX4PG7GTGEMQEPOUZA, seg_pos={0 0}, upload=439933]: Done uploading piece. err: <nil> Dec 26 12:45:40.769117454 (501h16m40.640200489s) [noise=true , address=145.79.227.32:4834 , piece_id=6EAMXXCFN7SLLPHYX4TXF45M4J6TRLLRGEFZVNVSEHXDTXZC42LA, seg_pos={0 0}, upload=439933]: Done uploading piece. err: <nil> Dec 26 12:45:41.010937561 (501h16m40.882020596s) [noise=true , address=45.91.116.9:4262 , piece_id=NC57EI6ZMEQTBVDGZCCAVSCYMZAYUGPRNYDCAL67VIC4JPQOZZ6Q, seg_pos={0 0}, upload=439933]: Done uploading piece. err: <nil> Dec 26 12:45:41.059442423 (501h16m40.930525458s) [noise=true , address=77.90.31.14:4663 , piece_id=C24I5YETHX7QXOGEGTLFPIT7WJ4I5KCTDRC4GLNVJ5AI2HMSOZWQ, seg_pos={0 0}, upload=439933]: Done uploading piece. err: <nil> Dec 26 12:45:41.301071478 (501h16m41.172154533s) [noise=true , address=45.91.116.5:4299 , piece_id=ITEU2CA4H3MXUZMI72O4BSWTN47CBX6UK3IV2P3YJUNPPEXU7SLA, seg_pos={0 0}, upload=439933]: Done uploading piece. err: <nil> Dec 26 12:45:41.397894325 (501h16m41.26897736s) [noise=true , address=77.90.31.10:4519 , piece_id=2AZN5NPX7JMYCMRGJSEM4B3NOKF4Y4KAJCNU7NMFPIHEXWS5JBBQ, seg_pos={0 0}, upload=439933]: Done uploading piece. err: <nil> Dec 26 12:45:41.397914042 (501h16m41.268997078s) [seg_pos={0 0}, upload=439933]: Segment reached optimal threshold of 13 pieces. Dec 26 12:45:41.398126918 (501h16m41.269209953s) [noise=true , address=45.91.116.12:4228 , piece_id=XVBGLE3BIPGLXH2F3FXAHXSAFSRW5R4AWNPNINIIAHXDWJ4Z4MYA, seg_pos={0 0}, upload=439933]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:59016->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:59016->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59016->45.91.116.12:4228: use of closed network connection Dec 26 12:45:41.398134352 (501h16m41.269217387s) [noise=true , address=45.91.116.12:4228 , piece_id=XVBGLE3BIPGLXH2F3FXAHXSAFSRW5R4AWNPNINIIAHXDWJ4Z4MYA, seg_pos={0 0}, upload=439933]: Long tail cancellation due to optimal threshold Dec 26 12:45:41.398245023 (501h16m41.269328068s) [noise=true , address=45.91.116.13:4153 , piece_id=L7EC4Z62EWI6C4BTN7G7RZLRNUAHDVY6FLPVXB6MCIXMXDVRJDDA, seg_pos={0 0}, upload=439933]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:35936->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:35936->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35936->45.91.116.13:4153: use of closed network connection Dec 26 12:45:41.398274619 (501h16m41.269357654s) [noise=true , address=45.91.116.13:4153 , piece_id=L7EC4Z62EWI6C4BTN7G7RZLRNUAHDVY6FLPVXB6MCIXMXDVRJDDA, seg_pos={0 0}, upload=439933]: Long tail cancellation due to optimal threshold Dec 26 12:45:41.398284669 (501h16m41.269367714s) [noise=true , address=45.91.116.14:4190 , piece_id=PHUFBC3OWAZOJMRGSWW56ZG4CFTTBUQP22TZRTTQYL2F7VJADWUA, seg_pos={0 0}, upload=439933]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 26 12:45:41.398291692 (501h16m41.269374737s) [noise=true , address=45.91.116.14:4190 , piece_id=PHUFBC3OWAZOJMRGSWW56ZG4CFTTBUQP22TZRTTQYL2F7VJADWUA, seg_pos={0 0}, upload=439933]: Long tail cancellation due to optimal threshold Dec 26 12:45:41.398371283 (501h16m41.269454328s) [seg_pos={0 0}, upload=439933]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:45:41.398544663 (501h16m41.269627698s) [upload=439933]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:45:41.398549813 (501h16m41.269632848s) [upload=439933]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:45:41.694499974 (501h16m41.565583019s) [upload=439933]: Done uploading segments... Dec 26 12:45:41.694523549 (501h16m41.565606584s) [upload=439933]: Upload finished. err: <nil> Duration: 30.004999911s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [.....X..X......X.......XXX...X.....X.......X.XX......XXX..XX] duration: 5.001176545s trace: https://jaeger.dev.storj.io/trace/6278b0df4abaea FAIL (2025-12-26 12:45:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001203718s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................................X....................X.XXX.] duration: 54.552452535s trace: https://jaeger.dev.storj.io/trace/399fe67f25bace88 FAIL (2025-12-26 12:45:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, Piece ID: 5BGLHPSJBADM4I66EOGEJYRLZFFPAHJ53MHVFDRSX24WUYY7YI2A) context deadline exceeded; piecestore: (Node ID: 12ZH3hS6xJp29U2RWAN9p8kxNdkSUQr1E58HzcFei48fUbNzFvW, Piece ID: FA6GAMWZ5EU23JWHVCFUY5ZJW6465F6ATA6RG4PH2H36URJVT65A) context deadline exceeded; piecestore: (Node ID: 1ftkFznfF2NSYrtgqUm3hgcyfwaZtWQfh2nDBEawxPh4SecemT, Piece ID: RSKD22V3Q23WFN4OVBYBLTSCOZJ2PXTVUSF4BIVIRSI6D2NC5YEA) context deadline exceeded ======LOGS====== EUHA, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.664140644 (501h14m37.535223679s) [noise=true , address=77.90.31.18:4822 , piece_id=2AL5F3CR5TACQXAHM7HGTTIHV6EUVF7P4NFVXO6V7ORAKBQM6QGA, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.664597445 (501h14m37.53568048s) [noise=true , address=145.79.227.34:4931 , piece_id=NMROTFKKS7GAMJZCZYLOIFDIUG6VP4D5W6HJ6B7DYVORMIQW5U6Q, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.665124648 (501h14m37.536207684s) [noise=true , address=77.90.31.21:4905 , piece_id=JV5W42QWWKR7ZZTRCEM2OVUIPWXFPTVMCDJ7NMF4YCPCJTB5WS5Q, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.665665368 (501h14m37.536748413s) [noise=true , address=145.79.227.44:4203 , piece_id=2KUGM74WCNULTSMYND6F32TXTWKPAP7UNNVRH3277TD2XUX3CSYA, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.666319684 (501h14m37.537402729s) [noise=true , address=77.90.31.13:4637 , piece_id=3RRZLEC4RZVM2UBMMIIJTQGQA5LHTQUFUUHKZFI57IU4MWC6ZJMQ, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.666533631 (501h14m37.537616676s) [noise=true , address=145.79.227.29:4965 , piece_id=D7YPJAELU2EGK72LINU773ZMFPMXRCH3YTDTAI3VSXDZXH5JCFFQ, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.667061025 (501h14m37.53814406s) [noise=true , address=145.79.227.27:4707 , piece_id=V3N5RRIZWZX72O4AZM2EQG34RU4FNWGBXO7JJSMYRQIBVI4UQ4RA, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.667575655 (501h14m37.53865869s) [noise=true , address=77.90.31.14:4686 , piece_id=5BGLHPSJBADM4I66EOGEJYRLZFFPAHJ53MHVFDRSX24WUYY7YI2A, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.668173774 (501h14m37.539256819s) [noise=true , address=77.90.31.12:4601 , piece_id=44WEMBXHWAGSGPPK5UQ2D2BL7OIAXYKNX2VV4Z26KZ6RIYQKSTFQ, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.668191278 (501h14m37.539274313s) [noise=true , address=45.91.116.14:4220 , piece_id=FA6GAMWZ5EU23JWHVCFUY5ZJW6465F6ATA6RG4PH2H36URJVT65A, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.668671292 (501h14m37.539754327s) [noise=true , address=145.79.227.40:4080 , piece_id=UQSPVT5RFKOHTN5TGLZWL5RYFR5TTR4FCUHNFZFCEDDZCROOBQVQ, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.669122792 (501h14m37.540205827s) [noise=true , address=77.90.31.17:4800 , piece_id=K72DLMFURND3TDQF3LRBH2D3BKGQBK73VKUGCHBWWENQUA3PB2OA, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.669621822 (501h14m37.540704867s) [noise=true , address=77.90.31.16:4764 , piece_id=RSKD22V3Q23WFN4OVBYBLTSCOZJ2PXTVUSF4BIVIRSI6D2NC5YEA, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.671130053 (501h14m37.542213088s) [noise=true , address=145.79.227.30:4767 , piece_id=64OMR2SYEU3TQYGEIBTLE3UAMWEA2C2R3XNWTCX2FKF3STLWD2YA, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.671287944 (501h14m37.542370979s) [noise=true , address=64.49.13.169:4706 , piece_id=LWAI7OPYOOSEJTWJU3WAJ4RRYOLBIHGGZUF7DHQR23ZKSGCDWQFA, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:39.141878555 (501h14m39.012961591s) [upload=439902]: Next returned nil segment Dec 26 12:43:39.141893735 (501h14m39.01297678s) [upload=439902]: Waiting for error group managing segments... Dec 26 12:43:39.537095507 (501h14m39.408178553s) [noise=true , address=145.79.227.24:4584 , piece_id=D5QC26FDHQ5TQBGFFWJUOOWUGIZSAYJS5KOOJFTAD6S7FW4YTFGA, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:39.544128439 (501h14m39.415211484s) [noise=true , address=145.79.227.37:4570 , piece_id=E6XQYXQ4JXMIJWF6J4DCJ4P6565N7JBWTLQ4GRFO6FYEMYPVATLQ, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:39.562038233 (501h14m39.433121278s) [noise=true , address=145.79.227.29:4965 , piece_id=D7YPJAELU2EGK72LINU773ZMFPMXRCH3YTDTAI3VSXDZXH5JCFFQ, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:39.577117587 (501h14m39.448200622s) [noise=true , address=145.79.227.27:4707 , piece_id=V3N5RRIZWZX72O4AZM2EQG34RU4FNWGBXO7JJSMYRQIBVI4UQ4RA, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:39.577340131 (501h14m39.448423166s) [noise=true , address=145.79.227.34:4931 , piece_id=NMROTFKKS7GAMJZCZYLOIFDIUG6VP4D5W6HJ6B7DYVORMIQW5U6Q, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:39.705570193 (501h14m39.576653238s) [noise=true , address=77.90.31.14:4686 , piece_id=5BGLHPSJBADM4I66EOGEJYRLZFFPAHJ53MHVFDRSX24WUYY7YI2A, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:39.772478088 (501h14m39.643561133s) [noise=true , address=45.91.116.14:4220 , piece_id=FA6GAMWZ5EU23JWHVCFUY5ZJW6465F6ATA6RG4PH2H36URJVT65A, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:39.831903704 (501h14m39.702986749s) [noise=true , address=77.90.31.13:4637 , piece_id=3RRZLEC4RZVM2UBMMIIJTQGQA5LHTQUFUUHKZFI57IU4MWC6ZJMQ, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:39.832432251 (501h14m39.703515286s) [noise=true , address=77.90.31.18:4822 , piece_id=2AL5F3CR5TACQXAHM7HGTTIHV6EUVF7P4NFVXO6V7ORAKBQM6QGA, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:39.844820801 (501h14m39.715903846s) [noise=true , address=77.90.31.16:4764 , piece_id=RSKD22V3Q23WFN4OVBYBLTSCOZJ2PXTVUSF4BIVIRSI6D2NC5YEA, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:39.854406263 (501h14m39.725489298s) [noise=true , address=77.90.31.15:4726 , piece_id=I47IGY4BJHX42ODE4P3CZVWNA5EBT7XKIBAUYUOAKXOWBCCUEUHA, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:40.410085534 (501h14m40.281168579s) [noise=true , address=145.79.227.44:4203 , piece_id=2KUGM74WCNULTSMYND6F32TXTWKPAP7UNNVRH3277TD2XUX3CSYA, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:40.488659412 (501h14m40.359742447s) [noise=true , address=145.79.227.30:4767 , piece_id=64OMR2SYEU3TQYGEIBTLE3UAMWEA2C2R3XNWTCX2FKF3STLWD2YA, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:40.684298553 (501h14m40.555381598s) [noise=true , address=77.90.31.17:4800 , piece_id=K72DLMFURND3TDQF3LRBH2D3BKGQBK73VKUGCHBWWENQUA3PB2OA, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:40.684320956 (501h14m40.555404001s) [seg_pos={0 0}, upload=439902]: Segment reached optimal threshold of 14 pieces. Dec 26 12:43:40.684595939 (501h14m40.555678994s) [noise=true , address=77.90.31.21:4905 , piece_id=JV5W42QWWKR7ZZTRCEM2OVUIPWXFPTVMCDJ7NMF4YCPCJTB5WS5Q, seg_pos={0 0}, upload=439902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK): protocol: expected piece hash; context canceled; EOF Dec 26 12:43:40.684606179 (501h14m40.555689214s) [noise=true , address=77.90.31.21:4905 , piece_id=JV5W42QWWKR7ZZTRCEM2OVUIPWXFPTVMCDJ7NMF4YCPCJTB5WS5Q, seg_pos={0 0}, upload=439902]: Long tail cancellation due to optimal threshold Dec 26 12:43:40.684627660 (501h14m40.555710715s) [noise=true , address=64.49.13.169:4706 , piece_id=LWAI7OPYOOSEJTWJU3WAJ4RRYOLBIHGGZUF7DHQR23ZKSGCDWQFA, seg_pos={0 0}, upload=439902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:41264->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:41264->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41264->64.49.13.169:4706: use of closed network connection Dec 26 12:43:40.684637739 (501h14m40.555720784s) [noise=true , address=64.49.13.169:4706 , piece_id=LWAI7OPYOOSEJTWJU3WAJ4RRYOLBIHGGZUF7DHQR23ZKSGCDWQFA, seg_pos={0 0}, upload=439902]: Long tail cancellation due to optimal threshold Dec 26 12:43:40.684840836 (501h14m40.555923891s) [noise=true , address=77.90.31.12:4601 , piece_id=44WEMBXHWAGSGPPK5UQ2D2BL7OIAXYKNX2VV4Z26KZ6RIYQKSTFQ, seg_pos={0 0}, upload=439902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pQPpDBX1nYAGc961a72T8NyFRc8yLMKfvYBqwuNBYQetzQZL2): protocol: read tcp 188.166.225.170:56438->77.90.31.12:4601: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 12:43:40.684849853 (501h14m40.555932898s) [noise=true , address=77.90.31.12:4601 , piece_id=44WEMBXHWAGSGPPK5UQ2D2BL7OIAXYKNX2VV4Z26KZ6RIYQKSTFQ, seg_pos={0 0}, upload=439902]: Long tail cancellation due to optimal threshold Dec 26 12:43:40.684993706 (501h14m40.556076741s) [noise=true , address=145.79.227.40:4080 , piece_id=UQSPVT5RFKOHTN5TGLZWL5RYFR5TTR4FCUHNFZFCEDDZCROOBQVQ, seg_pos={0 0}, upload=439902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WXMdaUQnFnXfayz6CkmsK1wuLhWSpXjQQt2kZyVTVnv3qL44C): protocol: expected piece hash; context canceled; EOF Dec 26 12:43:40.685002634 (501h14m40.556085679s) [noise=true , address=145.79.227.40:4080 , piece_id=UQSPVT5RFKOHTN5TGLZWL5RYFR5TTR4FCUHNFZFCEDDZCROOBQVQ, seg_pos={0 0}, upload=439902]: Long tail cancellation due to optimal threshold Dec 26 12:43:40.685024215 (501h14m40.55610725s) [seg_pos={0 0}, upload=439902]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 12:43:40.685126900 (501h14m40.556209945s) [upload=439902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:43:40.685134925 (501h14m40.55621796s) [upload=439902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:43:40.978747536 (501h14m40.849830581s) [upload=439902]: Done uploading segments... Dec 26 12:43:40.978770249 (501h14m40.849853294s) [upload=439902]: Upload finished. err: <nil> Duration: 54.552482653s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X...X.X....X..X...X..XX.....XXXX.....XXX....XX....X..X......] duration: 5.960414473s trace: https://jaeger.dev.storj.io/trace/f97f56868e2aa7c FAIL (2025-12-26 12:45:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: 2AQIKR4J7MCJXU67OL5ANUJ5MKHPDOTDX6GAQWNEORGKBWF7JC3A) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: 4MHNP5IJGVLZNFVEK7VX5CYMX33STTZ5CB736Z3Q4W5JAWKAPL2Q) context deadline exceeded ======LOGS====== Dec 26 12:41:48.442712118 (501h12m48.313795153s) [upload=439875]: Starting upload Dec 26 12:41:48.442786820 (501h12m48.313869885s) [upload=439875]: Done starting upload Dec 26 12:41:48.442961412 (501h12m48.314044447s) [upload=439875]: Uploading segments... Dec 26 12:41:48.443099966 (501h12m48.314183011s) [upload=439875]: Got next segment. Inline: false Dec 26 12:41:48.443286161 (501h12m48.314369206s) [upload=439875]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:41:48.443293645 (501h12m48.31437669s) [upload=439875]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:41:48.691376859 (501h12m48.562459904s) [seg_pos={0 0}, upload=439875]: Begin upload segment... Dec 26 12:41:48.691439108 (501h12m48.562522143s) [seg_pos={0 0}, upload=439875]: Done begin upload segment. Dec 26 12:41:48.691460008 (501h12m48.562543063s) [noise=true , address=45.91.116.8:4048 , piece_id=WM5CKKU336UI5K7JWSIYDS6V36AST7BYYGBNXGMIJYHBRSCTV7CA, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.691489002 (501h12m48.562572047s) [noise=true , address=145.79.227.11:4730 , piece_id=YQKC7KHSNITBZP6VPLES4I6HKQSPQWZQENYTOTF625W2AQPA57SA, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.691613129 (501h12m48.562696164s) [noise=true , address=145.79.227.52:4446 , piece_id=NQSLB7SPF24I3Q3OYY5M3GUHGPLENYQINOBNFP37WVEUPBSVE6TA, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.692179948 (501h12m48.563262993s) [noise=true , address=45.91.116.7:4011 , piece_id=4HYRSFIQOJQGESWRM2HLMYG6RDZHDEM2I6KZ4FZNOBHAKV24OLLQ, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.692193374 (501h12m48.563276419s) [noise=true , address=145.79.227.45:4230 , piece_id=WNEQ4D2ZEPXLIPPD7NFRSCDZJAQREYN2MYUCTVSUI3IWKZLECNFA, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.692668809 (501h12m48.563751854s) [noise=true , address=45.91.116.6:4976 , piece_id=4MHNP5IJGVLZNFVEK7VX5CYMX33STTZ5CB736Z3Q4W5JAWKAPL2Q, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.692706250 (501h12m48.563789285s) [noise=true , address=145.79.227.31:4799 , piece_id=CBATLHCVE6HI77XBXUBVI4AAFQXZJUUEWKMJ6LDLDQ4MHM5VIWSA, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.693084140 (501h12m48.564167185s) [noise=true , address=45.91.116.9:4264 , piece_id=2AQIKR4J7MCJXU67OL5ANUJ5MKHPDOTDX6GAQWNEORGKBWF7JC3A, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.693090893 (501h12m48.564173938s) [noise=true , address=145.79.227.12:4765 , piece_id=2ECTKX35BA2OCYSMI7JRFLGW4HZ5XJNNIDCEUDLSRZPVJLCJSKDA, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.693124386 (501h12m48.564207431s) [noise=true , address=45.91.116.14:4192 , piece_id=H6OJ5UZO7D5PHMBZVLSYI6XTV5GKNGLL2EQNM2WOA36RGHW6CVOA, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.693494481 (501h12m48.564577516s) [noise=true , address=145.79.227.37:4566 , piece_id=EGL5OXQLMH7IBQABZNAUQREEK57CWAP7D2ZAZI7ARXJT2JC5PM5A, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.693930231 (501h12m48.565013276s) [noise=true , address=145.79.227.3:4149 , piece_id=TPOZTDBJEVKZB6SE4DMKMSI7VZQDCO2PZTIPXOYOAMCUCIQTHKQA, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.694324161 (501h12m48.565407197s) [noise=true , address=145.79.227.46:4603 , piece_id=TJZ4R3RBT2ZCB5JCEAF6HGS5D7CFCLBK2JBYFON4XJN7TYTG2NNA, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.694680870 (501h12m48.565763916s) [noise=true , address=145.79.227.41:4086 , piece_id=YAPMPUFKJB4CNXSC6SENO23YGOU5677ZWNSFGZU2FNDZWXFICU7Q, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.694798895 (501h12m48.56588194s) [noise=true , address=145.79.227.36:4517 , piece_id=DE5JLBJWHIHXAFSGKELLYALASXE2PUUFY6VDE6TJKAQGM2UPYZTQ, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.695062958 (501h12m48.566145993s) [noise=true , address=145.79.227.32:4834 , piece_id=3MU6IJPI7BFU7WKCV2JNOMAINNABUMBKNYEBGSCJBBMTM7YXJK6A, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:50.410149874 (501h12m50.281232919s) [upload=439875]: Next returned nil segment Dec 26 12:41:50.410165523 (501h12m50.281248568s) [upload=439875]: Waiting for error group managing segments... Dec 26 12:41:50.690475803 (501h12m50.561558838s) [noise=true , address=145.79.227.11:4730 , piece_id=YQKC7KHSNITBZP6VPLES4I6HKQSPQWZQENYTOTF625W2AQPA57SA, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.699134751 (501h12m50.570217796s) [noise=true , address=145.79.227.45:4230 , piece_id=WNEQ4D2ZEPXLIPPD7NFRSCDZJAQREYN2MYUCTVSUI3IWKZLECNFA, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.699496050 (501h12m50.570579095s) [noise=true , address=145.79.227.52:4446 , piece_id=NQSLB7SPF24I3Q3OYY5M3GUHGPLENYQINOBNFP37WVEUPBSVE6TA, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.705921484 (501h12m50.577004519s) [noise=true , address=145.79.227.3:4149 , piece_id=TPOZTDBJEVKZB6SE4DMKMSI7VZQDCO2PZTIPXOYOAMCUCIQTHKQA, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.706349649 (501h12m50.577432684s) [noise=true , address=145.79.227.37:4566 , piece_id=EGL5OXQLMH7IBQABZNAUQREEK57CWAP7D2ZAZI7ARXJT2JC5PM5A, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.717218657 (501h12m50.588301692s) [noise=true , address=145.79.227.31:4799 , piece_id=CBATLHCVE6HI77XBXUBVI4AAFQXZJUUEWKMJ6LDLDQ4MHM5VIWSA, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.717630781 (501h12m50.588713817s) [noise=true , address=145.79.227.32:4834 , piece_id=3MU6IJPI7BFU7WKCV2JNOMAINNABUMBKNYEBGSCJBBMTM7YXJK6A, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.717969365 (501h12m50.589052401s) [noise=true , address=145.79.227.41:4086 , piece_id=YAPMPUFKJB4CNXSC6SENO23YGOU5677ZWNSFGZU2FNDZWXFICU7Q, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.720454037 (501h12m50.591537082s) [noise=true , address=145.79.227.12:4765 , piece_id=2ECTKX35BA2OCYSMI7JRFLGW4HZ5XJNNIDCEUDLSRZPVJLCJSKDA, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.722154334 (501h12m50.593237379s) [noise=true , address=145.79.227.46:4603 , piece_id=TJZ4R3RBT2ZCB5JCEAF6HGS5D7CFCLBK2JBYFON4XJN7TYTG2NNA, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.728157947 (501h12m50.599240982s) [noise=true , address=145.79.227.36:4517 , piece_id=DE5JLBJWHIHXAFSGKELLYALASXE2PUUFY6VDE6TJKAQGM2UPYZTQ, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.940256359 (501h12m50.811339404s) [noise=true , address=45.91.116.6:4976 , piece_id=4MHNP5IJGVLZNFVEK7VX5CYMX33STTZ5CB736Z3Q4W5JAWKAPL2Q, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:51.054849925 (501h12m50.92593296s) [noise=true , address=45.91.116.9:4264 , piece_id=2AQIKR4J7MCJXU67OL5ANUJ5MKHPDOTDX6GAQWNEORGKBWF7JC3A, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:51.054869883 (501h12m50.925952918s) [seg_pos={0 0}, upload=439875]: Segment reached optimal threshold of 13 pieces. Dec 26 12:41:51.055140669 (501h12m50.926223704s) [noise=true , address=45.91.116.8:4048 , piece_id=WM5CKKU336UI5K7JWSIYDS6V36AST7BYYGBNXGMIJYHBRSCTV7CA, seg_pos={0 0}, upload=439875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 26 12:41:51.055148013 (501h12m50.926231048s) [noise=true , address=45.91.116.8:4048 , piece_id=WM5CKKU336UI5K7JWSIYDS6V36AST7BYYGBNXGMIJYHBRSCTV7CA, seg_pos={0 0}, upload=439875]: Long tail cancellation due to optimal threshold Dec 26 12:41:51.055305071 (501h12m50.926388106s) [noise=true , address=45.91.116.14:4192 , piece_id=H6OJ5UZO7D5PHMBZVLSYI6XTV5GKNGLL2EQNM2WOA36RGHW6CVOA, seg_pos={0 0}, upload=439875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 26 12:41:51.055311344 (501h12m50.926394379s) [noise=true , address=45.91.116.14:4192 , piece_id=H6OJ5UZO7D5PHMBZVLSYI6XTV5GKNGLL2EQNM2WOA36RGHW6CVOA, seg_pos={0 0}, upload=439875]: Long tail cancellation due to optimal threshold Dec 26 12:41:51.055332804 (501h12m50.926415839s) [noise=true , address=45.91.116.7:4011 , piece_id=4HYRSFIQOJQGESWRM2HLMYG6RDZHDEM2I6KZ4FZNOBHAKV24OLLQ, seg_pos={0 0}, upload=439875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 12:41:51.055351250 (501h12m50.926434295s) [noise=true , address=45.91.116.7:4011 , piece_id=4HYRSFIQOJQGESWRM2HLMYG6RDZHDEM2I6KZ4FZNOBHAKV24OLLQ, seg_pos={0 0}, upload=439875]: Long tail cancellation due to optimal threshold Dec 26 12:41:51.055384263 (501h12m50.926467308s) [seg_pos={0 0}, upload=439875]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:41:51.055458073 (501h12m50.926541108s) [upload=439875]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:41:51.055464826 (501h12m50.926547871s) [upload=439875]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:41:51.345060146 (501h12m51.216143191s) [upload=439875]: Done uploading segments... Dec 26 12:41:51.345082800 (501h12m51.216165855s) [upload=439875]: Upload finished. err: <nil> Duration: 5.960446163s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [......X..X......X.......XXX...X.....X.......X.XX......XXX..X] duration: 5.001739297s trace: https://jaeger.dev.storj.io/trace/40d611c3c5800137 FAIL (2025-12-26 12:44:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001766748s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [XX...X.X....X..X...X..XX.....XXXX.....XXX....XX....X..X.....] duration: 5.798542419s trace: https://jaeger.dev.storj.io/trace/f97f56868e2aa7c FAIL (2025-12-26 12:44:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: 2AQIKR4J7MCJXU67OL5ANUJ5MKHPDOTDX6GAQWNEORGKBWF7JC3A) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: 4MHNP5IJGVLZNFVEK7VX5CYMX33STTZ5CB736Z3Q4W5JAWKAPL2Q) context deadline exceeded ======LOGS====== Dec 26 12:41:48.442712118 (501h12m48.313795153s) [upload=439875]: Starting upload Dec 26 12:41:48.442786820 (501h12m48.313869885s) [upload=439875]: Done starting upload Dec 26 12:41:48.442961412 (501h12m48.314044447s) [upload=439875]: Uploading segments... Dec 26 12:41:48.443099966 (501h12m48.314183011s) [upload=439875]: Got next segment. Inline: false Dec 26 12:41:48.443286161 (501h12m48.314369206s) [upload=439875]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:41:48.443293645 (501h12m48.31437669s) [upload=439875]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:41:48.691376859 (501h12m48.562459904s) [seg_pos={0 0}, upload=439875]: Begin upload segment... Dec 26 12:41:48.691439108 (501h12m48.562522143s) [seg_pos={0 0}, upload=439875]: Done begin upload segment. Dec 26 12:41:48.691460008 (501h12m48.562543063s) [noise=true , address=45.91.116.8:4048 , piece_id=WM5CKKU336UI5K7JWSIYDS6V36AST7BYYGBNXGMIJYHBRSCTV7CA, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.691489002 (501h12m48.562572047s) [noise=true , address=145.79.227.11:4730 , piece_id=YQKC7KHSNITBZP6VPLES4I6HKQSPQWZQENYTOTF625W2AQPA57SA, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.691613129 (501h12m48.562696164s) [noise=true , address=145.79.227.52:4446 , piece_id=NQSLB7SPF24I3Q3OYY5M3GUHGPLENYQINOBNFP37WVEUPBSVE6TA, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.692179948 (501h12m48.563262993s) [noise=true , address=45.91.116.7:4011 , piece_id=4HYRSFIQOJQGESWRM2HLMYG6RDZHDEM2I6KZ4FZNOBHAKV24OLLQ, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.692193374 (501h12m48.563276419s) [noise=true , address=145.79.227.45:4230 , piece_id=WNEQ4D2ZEPXLIPPD7NFRSCDZJAQREYN2MYUCTVSUI3IWKZLECNFA, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.692668809 (501h12m48.563751854s) [noise=true , address=45.91.116.6:4976 , piece_id=4MHNP5IJGVLZNFVEK7VX5CYMX33STTZ5CB736Z3Q4W5JAWKAPL2Q, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.692706250 (501h12m48.563789285s) [noise=true , address=145.79.227.31:4799 , piece_id=CBATLHCVE6HI77XBXUBVI4AAFQXZJUUEWKMJ6LDLDQ4MHM5VIWSA, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.693084140 (501h12m48.564167185s) [noise=true , address=45.91.116.9:4264 , piece_id=2AQIKR4J7MCJXU67OL5ANUJ5MKHPDOTDX6GAQWNEORGKBWF7JC3A, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.693090893 (501h12m48.564173938s) [noise=true , address=145.79.227.12:4765 , piece_id=2ECTKX35BA2OCYSMI7JRFLGW4HZ5XJNNIDCEUDLSRZPVJLCJSKDA, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.693124386 (501h12m48.564207431s) [noise=true , address=45.91.116.14:4192 , piece_id=H6OJ5UZO7D5PHMBZVLSYI6XTV5GKNGLL2EQNM2WOA36RGHW6CVOA, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.693494481 (501h12m48.564577516s) [noise=true , address=145.79.227.37:4566 , piece_id=EGL5OXQLMH7IBQABZNAUQREEK57CWAP7D2ZAZI7ARXJT2JC5PM5A, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.693930231 (501h12m48.565013276s) [noise=true , address=145.79.227.3:4149 , piece_id=TPOZTDBJEVKZB6SE4DMKMSI7VZQDCO2PZTIPXOYOAMCUCIQTHKQA, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.694324161 (501h12m48.565407197s) [noise=true , address=145.79.227.46:4603 , piece_id=TJZ4R3RBT2ZCB5JCEAF6HGS5D7CFCLBK2JBYFON4XJN7TYTG2NNA, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.694680870 (501h12m48.565763916s) [noise=true , address=145.79.227.41:4086 , piece_id=YAPMPUFKJB4CNXSC6SENO23YGOU5677ZWNSFGZU2FNDZWXFICU7Q, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.694798895 (501h12m48.56588194s) [noise=true , address=145.79.227.36:4517 , piece_id=DE5JLBJWHIHXAFSGKELLYALASXE2PUUFY6VDE6TJKAQGM2UPYZTQ, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.695062958 (501h12m48.566145993s) [noise=true , address=145.79.227.32:4834 , piece_id=3MU6IJPI7BFU7WKCV2JNOMAINNABUMBKNYEBGSCJBBMTM7YXJK6A, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:50.410149874 (501h12m50.281232919s) [upload=439875]: Next returned nil segment Dec 26 12:41:50.410165523 (501h12m50.281248568s) [upload=439875]: Waiting for error group managing segments... Dec 26 12:41:50.690475803 (501h12m50.561558838s) [noise=true , address=145.79.227.11:4730 , piece_id=YQKC7KHSNITBZP6VPLES4I6HKQSPQWZQENYTOTF625W2AQPA57SA, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.699134751 (501h12m50.570217796s) [noise=true , address=145.79.227.45:4230 , piece_id=WNEQ4D2ZEPXLIPPD7NFRSCDZJAQREYN2MYUCTVSUI3IWKZLECNFA, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.699496050 (501h12m50.570579095s) [noise=true , address=145.79.227.52:4446 , piece_id=NQSLB7SPF24I3Q3OYY5M3GUHGPLENYQINOBNFP37WVEUPBSVE6TA, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.705921484 (501h12m50.577004519s) [noise=true , address=145.79.227.3:4149 , piece_id=TPOZTDBJEVKZB6SE4DMKMSI7VZQDCO2PZTIPXOYOAMCUCIQTHKQA, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.706349649 (501h12m50.577432684s) [noise=true , address=145.79.227.37:4566 , piece_id=EGL5OXQLMH7IBQABZNAUQREEK57CWAP7D2ZAZI7ARXJT2JC5PM5A, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.717218657 (501h12m50.588301692s) [noise=true , address=145.79.227.31:4799 , piece_id=CBATLHCVE6HI77XBXUBVI4AAFQXZJUUEWKMJ6LDLDQ4MHM5VIWSA, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.717630781 (501h12m50.588713817s) [noise=true , address=145.79.227.32:4834 , piece_id=3MU6IJPI7BFU7WKCV2JNOMAINNABUMBKNYEBGSCJBBMTM7YXJK6A, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.717969365 (501h12m50.589052401s) [noise=true , address=145.79.227.41:4086 , piece_id=YAPMPUFKJB4CNXSC6SENO23YGOU5677ZWNSFGZU2FNDZWXFICU7Q, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.720454037 (501h12m50.591537082s) [noise=true , address=145.79.227.12:4765 , piece_id=2ECTKX35BA2OCYSMI7JRFLGW4HZ5XJNNIDCEUDLSRZPVJLCJSKDA, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.722154334 (501h12m50.593237379s) [noise=true , address=145.79.227.46:4603 , piece_id=TJZ4R3RBT2ZCB5JCEAF6HGS5D7CFCLBK2JBYFON4XJN7TYTG2NNA, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.728157947 (501h12m50.599240982s) [noise=true , address=145.79.227.36:4517 , piece_id=DE5JLBJWHIHXAFSGKELLYALASXE2PUUFY6VDE6TJKAQGM2UPYZTQ, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.940256359 (501h12m50.811339404s) [noise=true , address=45.91.116.6:4976 , piece_id=4MHNP5IJGVLZNFVEK7VX5CYMX33STTZ5CB736Z3Q4W5JAWKAPL2Q, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:51.054849925 (501h12m50.92593296s) [noise=true , address=45.91.116.9:4264 , piece_id=2AQIKR4J7MCJXU67OL5ANUJ5MKHPDOTDX6GAQWNEORGKBWF7JC3A, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:51.054869883 (501h12m50.925952918s) [seg_pos={0 0}, upload=439875]: Segment reached optimal threshold of 13 pieces. Dec 26 12:41:51.055140669 (501h12m50.926223704s) [noise=true , address=45.91.116.8:4048 , piece_id=WM5CKKU336UI5K7JWSIYDS6V36AST7BYYGBNXGMIJYHBRSCTV7CA, seg_pos={0 0}, upload=439875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 26 12:41:51.055148013 (501h12m50.926231048s) [noise=true , address=45.91.116.8:4048 , piece_id=WM5CKKU336UI5K7JWSIYDS6V36AST7BYYGBNXGMIJYHBRSCTV7CA, seg_pos={0 0}, upload=439875]: Long tail cancellation due to optimal threshold Dec 26 12:41:51.055305071 (501h12m50.926388106s) [noise=true , address=45.91.116.14:4192 , piece_id=H6OJ5UZO7D5PHMBZVLSYI6XTV5GKNGLL2EQNM2WOA36RGHW6CVOA, seg_pos={0 0}, upload=439875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 26 12:41:51.055311344 (501h12m50.926394379s) [noise=true , address=45.91.116.14:4192 , piece_id=H6OJ5UZO7D5PHMBZVLSYI6XTV5GKNGLL2EQNM2WOA36RGHW6CVOA, seg_pos={0 0}, upload=439875]: Long tail cancellation due to optimal threshold Dec 26 12:41:51.055332804 (501h12m50.926415839s) [noise=true , address=45.91.116.7:4011 , piece_id=4HYRSFIQOJQGESWRM2HLMYG6RDZHDEM2I6KZ4FZNOBHAKV24OLLQ, seg_pos={0 0}, upload=439875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 12:41:51.055351250 (501h12m50.926434295s) [noise=true , address=45.91.116.7:4011 , piece_id=4HYRSFIQOJQGESWRM2HLMYG6RDZHDEM2I6KZ4FZNOBHAKV24OLLQ, seg_pos={0 0}, upload=439875]: Long tail cancellation due to optimal threshold Dec 26 12:41:51.055384263 (501h12m50.926467308s) [seg_pos={0 0}, upload=439875]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:41:51.055458073 (501h12m50.926541108s) [upload=439875]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:41:51.055464826 (501h12m50.926547871s) [upload=439875]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:41:51.345060146 (501h12m51.216143191s) [upload=439875]: Done uploading segments... Dec 26 12:41:51.345082800 (501h12m51.216165855s) [upload=439875]: Upload finished. err: <nil> Duration: 5.798570713s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................................X....................X.XXX] duration: 1m0.003159216s trace: https://jaeger.dev.storj.io/trace/399fe67f25bace88 FAIL (2025-12-26 12:44:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oiaCxBrU4T1tjfF18931ThYP82bK4V2qDG37nUUnMBm1EF1Cx, Piece ID: 5BGLHPSJBADM4I66EOGEJYRLZFFPAHJ53MHVFDRSX24WUYY7YI2A) context deadline exceeded; piecestore: (Node ID: 12ZH3hS6xJp29U2RWAN9p8kxNdkSUQr1E58HzcFei48fUbNzFvW, Piece ID: FA6GAMWZ5EU23JWHVCFUY5ZJW6465F6ATA6RG4PH2H36URJVT65A) context deadline exceeded; piecestore: (Node ID: 1ftkFznfF2NSYrtgqUm3hgcyfwaZtWQfh2nDBEawxPh4SecemT, Piece ID: RSKD22V3Q23WFN4OVBYBLTSCOZJ2PXTVUSF4BIVIRSI6D2NC5YEA) context deadline exceeded ======LOGS====== EUHA, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.664140644 (501h14m37.535223679s) [noise=true , address=77.90.31.18:4822 , piece_id=2AL5F3CR5TACQXAHM7HGTTIHV6EUVF7P4NFVXO6V7ORAKBQM6QGA, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.664597445 (501h14m37.53568048s) [noise=true , address=145.79.227.34:4931 , piece_id=NMROTFKKS7GAMJZCZYLOIFDIUG6VP4D5W6HJ6B7DYVORMIQW5U6Q, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.665124648 (501h14m37.536207684s) [noise=true , address=77.90.31.21:4905 , piece_id=JV5W42QWWKR7ZZTRCEM2OVUIPWXFPTVMCDJ7NMF4YCPCJTB5WS5Q, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.665665368 (501h14m37.536748413s) [noise=true , address=145.79.227.44:4203 , piece_id=2KUGM74WCNULTSMYND6F32TXTWKPAP7UNNVRH3277TD2XUX3CSYA, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.666319684 (501h14m37.537402729s) [noise=true , address=77.90.31.13:4637 , piece_id=3RRZLEC4RZVM2UBMMIIJTQGQA5LHTQUFUUHKZFI57IU4MWC6ZJMQ, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.666533631 (501h14m37.537616676s) [noise=true , address=145.79.227.29:4965 , piece_id=D7YPJAELU2EGK72LINU773ZMFPMXRCH3YTDTAI3VSXDZXH5JCFFQ, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.667061025 (501h14m37.53814406s) [noise=true , address=145.79.227.27:4707 , piece_id=V3N5RRIZWZX72O4AZM2EQG34RU4FNWGBXO7JJSMYRQIBVI4UQ4RA, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.667575655 (501h14m37.53865869s) [noise=true , address=77.90.31.14:4686 , piece_id=5BGLHPSJBADM4I66EOGEJYRLZFFPAHJ53MHVFDRSX24WUYY7YI2A, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.668173774 (501h14m37.539256819s) [noise=true , address=77.90.31.12:4601 , piece_id=44WEMBXHWAGSGPPK5UQ2D2BL7OIAXYKNX2VV4Z26KZ6RIYQKSTFQ, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.668191278 (501h14m37.539274313s) [noise=true , address=45.91.116.14:4220 , piece_id=FA6GAMWZ5EU23JWHVCFUY5ZJW6465F6ATA6RG4PH2H36URJVT65A, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.668671292 (501h14m37.539754327s) [noise=true , address=145.79.227.40:4080 , piece_id=UQSPVT5RFKOHTN5TGLZWL5RYFR5TTR4FCUHNFZFCEDDZCROOBQVQ, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.669122792 (501h14m37.540205827s) [noise=true , address=77.90.31.17:4800 , piece_id=K72DLMFURND3TDQF3LRBH2D3BKGQBK73VKUGCHBWWENQUA3PB2OA, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.669621822 (501h14m37.540704867s) [noise=true , address=77.90.31.16:4764 , piece_id=RSKD22V3Q23WFN4OVBYBLTSCOZJ2PXTVUSF4BIVIRSI6D2NC5YEA, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.671130053 (501h14m37.542213088s) [noise=true , address=145.79.227.30:4767 , piece_id=64OMR2SYEU3TQYGEIBTLE3UAMWEA2C2R3XNWTCX2FKF3STLWD2YA, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:37.671287944 (501h14m37.542370979s) [noise=true , address=64.49.13.169:4706 , piece_id=LWAI7OPYOOSEJTWJU3WAJ4RRYOLBIHGGZUF7DHQR23ZKSGCDWQFA, seg_pos={0 0}, upload=439902]: Uploading piece... Dec 26 12:43:39.141878555 (501h14m39.012961591s) [upload=439902]: Next returned nil segment Dec 26 12:43:39.141893735 (501h14m39.01297678s) [upload=439902]: Waiting for error group managing segments... Dec 26 12:43:39.537095507 (501h14m39.408178553s) [noise=true , address=145.79.227.24:4584 , piece_id=D5QC26FDHQ5TQBGFFWJUOOWUGIZSAYJS5KOOJFTAD6S7FW4YTFGA, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:39.544128439 (501h14m39.415211484s) [noise=true , address=145.79.227.37:4570 , piece_id=E6XQYXQ4JXMIJWF6J4DCJ4P6565N7JBWTLQ4GRFO6FYEMYPVATLQ, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:39.562038233 (501h14m39.433121278s) [noise=true , address=145.79.227.29:4965 , piece_id=D7YPJAELU2EGK72LINU773ZMFPMXRCH3YTDTAI3VSXDZXH5JCFFQ, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:39.577117587 (501h14m39.448200622s) [noise=true , address=145.79.227.27:4707 , piece_id=V3N5RRIZWZX72O4AZM2EQG34RU4FNWGBXO7JJSMYRQIBVI4UQ4RA, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:39.577340131 (501h14m39.448423166s) [noise=true , address=145.79.227.34:4931 , piece_id=NMROTFKKS7GAMJZCZYLOIFDIUG6VP4D5W6HJ6B7DYVORMIQW5U6Q, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:39.705570193 (501h14m39.576653238s) [noise=true , address=77.90.31.14:4686 , piece_id=5BGLHPSJBADM4I66EOGEJYRLZFFPAHJ53MHVFDRSX24WUYY7YI2A, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:39.772478088 (501h14m39.643561133s) [noise=true , address=45.91.116.14:4220 , piece_id=FA6GAMWZ5EU23JWHVCFUY5ZJW6465F6ATA6RG4PH2H36URJVT65A, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:39.831903704 (501h14m39.702986749s) [noise=true , address=77.90.31.13:4637 , piece_id=3RRZLEC4RZVM2UBMMIIJTQGQA5LHTQUFUUHKZFI57IU4MWC6ZJMQ, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:39.832432251 (501h14m39.703515286s) [noise=true , address=77.90.31.18:4822 , piece_id=2AL5F3CR5TACQXAHM7HGTTIHV6EUVF7P4NFVXO6V7ORAKBQM6QGA, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:39.844820801 (501h14m39.715903846s) [noise=true , address=77.90.31.16:4764 , piece_id=RSKD22V3Q23WFN4OVBYBLTSCOZJ2PXTVUSF4BIVIRSI6D2NC5YEA, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:39.854406263 (501h14m39.725489298s) [noise=true , address=77.90.31.15:4726 , piece_id=I47IGY4BJHX42ODE4P3CZVWNA5EBT7XKIBAUYUOAKXOWBCCUEUHA, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:40.410085534 (501h14m40.281168579s) [noise=true , address=145.79.227.44:4203 , piece_id=2KUGM74WCNULTSMYND6F32TXTWKPAP7UNNVRH3277TD2XUX3CSYA, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:40.488659412 (501h14m40.359742447s) [noise=true , address=145.79.227.30:4767 , piece_id=64OMR2SYEU3TQYGEIBTLE3UAMWEA2C2R3XNWTCX2FKF3STLWD2YA, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:40.684298553 (501h14m40.555381598s) [noise=true , address=77.90.31.17:4800 , piece_id=K72DLMFURND3TDQF3LRBH2D3BKGQBK73VKUGCHBWWENQUA3PB2OA, seg_pos={0 0}, upload=439902]: Done uploading piece. err: <nil> Dec 26 12:43:40.684320956 (501h14m40.555404001s) [seg_pos={0 0}, upload=439902]: Segment reached optimal threshold of 14 pieces. Dec 26 12:43:40.684595939 (501h14m40.555678994s) [noise=true , address=77.90.31.21:4905 , piece_id=JV5W42QWWKR7ZZTRCEM2OVUIPWXFPTVMCDJ7NMF4YCPCJTB5WS5Q, seg_pos={0 0}, upload=439902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK): protocol: expected piece hash; context canceled; EOF Dec 26 12:43:40.684606179 (501h14m40.555689214s) [noise=true , address=77.90.31.21:4905 , piece_id=JV5W42QWWKR7ZZTRCEM2OVUIPWXFPTVMCDJ7NMF4YCPCJTB5WS5Q, seg_pos={0 0}, upload=439902]: Long tail cancellation due to optimal threshold Dec 26 12:43:40.684627660 (501h14m40.555710715s) [noise=true , address=64.49.13.169:4706 , piece_id=LWAI7OPYOOSEJTWJU3WAJ4RRYOLBIHGGZUF7DHQR23ZKSGCDWQFA, seg_pos={0 0}, upload=439902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:41264->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:41264->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41264->64.49.13.169:4706: use of closed network connection Dec 26 12:43:40.684637739 (501h14m40.555720784s) [noise=true , address=64.49.13.169:4706 , piece_id=LWAI7OPYOOSEJTWJU3WAJ4RRYOLBIHGGZUF7DHQR23ZKSGCDWQFA, seg_pos={0 0}, upload=439902]: Long tail cancellation due to optimal threshold Dec 26 12:43:40.684840836 (501h14m40.555923891s) [noise=true , address=77.90.31.12:4601 , piece_id=44WEMBXHWAGSGPPK5UQ2D2BL7OIAXYKNX2VV4Z26KZ6RIYQKSTFQ, seg_pos={0 0}, upload=439902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pQPpDBX1nYAGc961a72T8NyFRc8yLMKfvYBqwuNBYQetzQZL2): protocol: read tcp 188.166.225.170:56438->77.90.31.12:4601: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 12:43:40.684849853 (501h14m40.555932898s) [noise=true , address=77.90.31.12:4601 , piece_id=44WEMBXHWAGSGPPK5UQ2D2BL7OIAXYKNX2VV4Z26KZ6RIYQKSTFQ, seg_pos={0 0}, upload=439902]: Long tail cancellation due to optimal threshold Dec 26 12:43:40.684993706 (501h14m40.556076741s) [noise=true , address=145.79.227.40:4080 , piece_id=UQSPVT5RFKOHTN5TGLZWL5RYFR5TTR4FCUHNFZFCEDDZCROOBQVQ, seg_pos={0 0}, upload=439902]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1WXMdaUQnFnXfayz6CkmsK1wuLhWSpXjQQt2kZyVTVnv3qL44C): protocol: expected piece hash; context canceled; EOF Dec 26 12:43:40.685002634 (501h14m40.556085679s) [noise=true , address=145.79.227.40:4080 , piece_id=UQSPVT5RFKOHTN5TGLZWL5RYFR5TTR4FCUHNFZFCEDDZCROOBQVQ, seg_pos={0 0}, upload=439902]: Long tail cancellation due to optimal threshold Dec 26 12:43:40.685024215 (501h14m40.55610725s) [seg_pos={0 0}, upload=439902]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 12:43:40.685126900 (501h14m40.556209945s) [upload=439902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:43:40.685134925 (501h14m40.55621796s) [upload=439902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:43:40.978747536 (501h14m40.849830581s) [upload=439902]: Done uploading segments... Dec 26 12:43:40.978770249 (501h14m40.849853294s) [upload=439902]: Upload finished. err: <nil> Duration: 1m0.003205425s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [.......X..X......X.......XXX...X.....X.......X.XX......XXX..] duration: 5.001122873s trace: https://jaeger.dev.storj.io/trace/3941b76d19abf99a FAIL (2025-12-26 12:44:30 UTC) orbiter warning: context deadline exceeded Duration: 5.001145878s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XXX...X.X....X..X...X..XX.....XXXX.....XXX....XX....X..X....] duration: 6.038254532s trace: https://jaeger.dev.storj.io/trace/f97f56868e2aa7c FAIL (2025-12-26 12:44:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: 2AQIKR4J7MCJXU67OL5ANUJ5MKHPDOTDX6GAQWNEORGKBWF7JC3A) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: 4MHNP5IJGVLZNFVEK7VX5CYMX33STTZ5CB736Z3Q4W5JAWKAPL2Q) context deadline exceeded ======LOGS====== Dec 26 12:41:48.442712118 (501h12m48.313795153s) [upload=439875]: Starting upload Dec 26 12:41:48.442786820 (501h12m48.313869885s) [upload=439875]: Done starting upload Dec 26 12:41:48.442961412 (501h12m48.314044447s) [upload=439875]: Uploading segments... Dec 26 12:41:48.443099966 (501h12m48.314183011s) [upload=439875]: Got next segment. Inline: false Dec 26 12:41:48.443286161 (501h12m48.314369206s) [upload=439875]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:41:48.443293645 (501h12m48.31437669s) [upload=439875]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:41:48.691376859 (501h12m48.562459904s) [seg_pos={0 0}, upload=439875]: Begin upload segment... Dec 26 12:41:48.691439108 (501h12m48.562522143s) [seg_pos={0 0}, upload=439875]: Done begin upload segment. Dec 26 12:41:48.691460008 (501h12m48.562543063s) [noise=true , address=45.91.116.8:4048 , piece_id=WM5CKKU336UI5K7JWSIYDS6V36AST7BYYGBNXGMIJYHBRSCTV7CA, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.691489002 (501h12m48.562572047s) [noise=true , address=145.79.227.11:4730 , piece_id=YQKC7KHSNITBZP6VPLES4I6HKQSPQWZQENYTOTF625W2AQPA57SA, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.691613129 (501h12m48.562696164s) [noise=true , address=145.79.227.52:4446 , piece_id=NQSLB7SPF24I3Q3OYY5M3GUHGPLENYQINOBNFP37WVEUPBSVE6TA, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.692179948 (501h12m48.563262993s) [noise=true , address=45.91.116.7:4011 , piece_id=4HYRSFIQOJQGESWRM2HLMYG6RDZHDEM2I6KZ4FZNOBHAKV24OLLQ, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.692193374 (501h12m48.563276419s) [noise=true , address=145.79.227.45:4230 , piece_id=WNEQ4D2ZEPXLIPPD7NFRSCDZJAQREYN2MYUCTVSUI3IWKZLECNFA, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.692668809 (501h12m48.563751854s) [noise=true , address=45.91.116.6:4976 , piece_id=4MHNP5IJGVLZNFVEK7VX5CYMX33STTZ5CB736Z3Q4W5JAWKAPL2Q, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.692706250 (501h12m48.563789285s) [noise=true , address=145.79.227.31:4799 , piece_id=CBATLHCVE6HI77XBXUBVI4AAFQXZJUUEWKMJ6LDLDQ4MHM5VIWSA, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.693084140 (501h12m48.564167185s) [noise=true , address=45.91.116.9:4264 , piece_id=2AQIKR4J7MCJXU67OL5ANUJ5MKHPDOTDX6GAQWNEORGKBWF7JC3A, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.693090893 (501h12m48.564173938s) [noise=true , address=145.79.227.12:4765 , piece_id=2ECTKX35BA2OCYSMI7JRFLGW4HZ5XJNNIDCEUDLSRZPVJLCJSKDA, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.693124386 (501h12m48.564207431s) [noise=true , address=45.91.116.14:4192 , piece_id=H6OJ5UZO7D5PHMBZVLSYI6XTV5GKNGLL2EQNM2WOA36RGHW6CVOA, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.693494481 (501h12m48.564577516s) [noise=true , address=145.79.227.37:4566 , piece_id=EGL5OXQLMH7IBQABZNAUQREEK57CWAP7D2ZAZI7ARXJT2JC5PM5A, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.693930231 (501h12m48.565013276s) [noise=true , address=145.79.227.3:4149 , piece_id=TPOZTDBJEVKZB6SE4DMKMSI7VZQDCO2PZTIPXOYOAMCUCIQTHKQA, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.694324161 (501h12m48.565407197s) [noise=true , address=145.79.227.46:4603 , piece_id=TJZ4R3RBT2ZCB5JCEAF6HGS5D7CFCLBK2JBYFON4XJN7TYTG2NNA, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.694680870 (501h12m48.565763916s) [noise=true , address=145.79.227.41:4086 , piece_id=YAPMPUFKJB4CNXSC6SENO23YGOU5677ZWNSFGZU2FNDZWXFICU7Q, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.694798895 (501h12m48.56588194s) [noise=true , address=145.79.227.36:4517 , piece_id=DE5JLBJWHIHXAFSGKELLYALASXE2PUUFY6VDE6TJKAQGM2UPYZTQ, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.695062958 (501h12m48.566145993s) [noise=true , address=145.79.227.32:4834 , piece_id=3MU6IJPI7BFU7WKCV2JNOMAINNABUMBKNYEBGSCJBBMTM7YXJK6A, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:50.410149874 (501h12m50.281232919s) [upload=439875]: Next returned nil segment Dec 26 12:41:50.410165523 (501h12m50.281248568s) [upload=439875]: Waiting for error group managing segments... Dec 26 12:41:50.690475803 (501h12m50.561558838s) [noise=true , address=145.79.227.11:4730 , piece_id=YQKC7KHSNITBZP6VPLES4I6HKQSPQWZQENYTOTF625W2AQPA57SA, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.699134751 (501h12m50.570217796s) [noise=true , address=145.79.227.45:4230 , piece_id=WNEQ4D2ZEPXLIPPD7NFRSCDZJAQREYN2MYUCTVSUI3IWKZLECNFA, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.699496050 (501h12m50.570579095s) [noise=true , address=145.79.227.52:4446 , piece_id=NQSLB7SPF24I3Q3OYY5M3GUHGPLENYQINOBNFP37WVEUPBSVE6TA, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.705921484 (501h12m50.577004519s) [noise=true , address=145.79.227.3:4149 , piece_id=TPOZTDBJEVKZB6SE4DMKMSI7VZQDCO2PZTIPXOYOAMCUCIQTHKQA, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.706349649 (501h12m50.577432684s) [noise=true , address=145.79.227.37:4566 , piece_id=EGL5OXQLMH7IBQABZNAUQREEK57CWAP7D2ZAZI7ARXJT2JC5PM5A, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.717218657 (501h12m50.588301692s) [noise=true , address=145.79.227.31:4799 , piece_id=CBATLHCVE6HI77XBXUBVI4AAFQXZJUUEWKMJ6LDLDQ4MHM5VIWSA, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.717630781 (501h12m50.588713817s) [noise=true , address=145.79.227.32:4834 , piece_id=3MU6IJPI7BFU7WKCV2JNOMAINNABUMBKNYEBGSCJBBMTM7YXJK6A, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.717969365 (501h12m50.589052401s) [noise=true , address=145.79.227.41:4086 , piece_id=YAPMPUFKJB4CNXSC6SENO23YGOU5677ZWNSFGZU2FNDZWXFICU7Q, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.720454037 (501h12m50.591537082s) [noise=true , address=145.79.227.12:4765 , piece_id=2ECTKX35BA2OCYSMI7JRFLGW4HZ5XJNNIDCEUDLSRZPVJLCJSKDA, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.722154334 (501h12m50.593237379s) [noise=true , address=145.79.227.46:4603 , piece_id=TJZ4R3RBT2ZCB5JCEAF6HGS5D7CFCLBK2JBYFON4XJN7TYTG2NNA, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.728157947 (501h12m50.599240982s) [noise=true , address=145.79.227.36:4517 , piece_id=DE5JLBJWHIHXAFSGKELLYALASXE2PUUFY6VDE6TJKAQGM2UPYZTQ, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.940256359 (501h12m50.811339404s) [noise=true , address=45.91.116.6:4976 , piece_id=4MHNP5IJGVLZNFVEK7VX5CYMX33STTZ5CB736Z3Q4W5JAWKAPL2Q, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:51.054849925 (501h12m50.92593296s) [noise=true , address=45.91.116.9:4264 , piece_id=2AQIKR4J7MCJXU67OL5ANUJ5MKHPDOTDX6GAQWNEORGKBWF7JC3A, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:51.054869883 (501h12m50.925952918s) [seg_pos={0 0}, upload=439875]: Segment reached optimal threshold of 13 pieces. Dec 26 12:41:51.055140669 (501h12m50.926223704s) [noise=true , address=45.91.116.8:4048 , piece_id=WM5CKKU336UI5K7JWSIYDS6V36AST7BYYGBNXGMIJYHBRSCTV7CA, seg_pos={0 0}, upload=439875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 26 12:41:51.055148013 (501h12m50.926231048s) [noise=true , address=45.91.116.8:4048 , piece_id=WM5CKKU336UI5K7JWSIYDS6V36AST7BYYGBNXGMIJYHBRSCTV7CA, seg_pos={0 0}, upload=439875]: Long tail cancellation due to optimal threshold Dec 26 12:41:51.055305071 (501h12m50.926388106s) [noise=true , address=45.91.116.14:4192 , piece_id=H6OJ5UZO7D5PHMBZVLSYI6XTV5GKNGLL2EQNM2WOA36RGHW6CVOA, seg_pos={0 0}, upload=439875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 26 12:41:51.055311344 (501h12m50.926394379s) [noise=true , address=45.91.116.14:4192 , piece_id=H6OJ5UZO7D5PHMBZVLSYI6XTV5GKNGLL2EQNM2WOA36RGHW6CVOA, seg_pos={0 0}, upload=439875]: Long tail cancellation due to optimal threshold Dec 26 12:41:51.055332804 (501h12m50.926415839s) [noise=true , address=45.91.116.7:4011 , piece_id=4HYRSFIQOJQGESWRM2HLMYG6RDZHDEM2I6KZ4FZNOBHAKV24OLLQ, seg_pos={0 0}, upload=439875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 12:41:51.055351250 (501h12m50.926434295s) [noise=true , address=45.91.116.7:4011 , piece_id=4HYRSFIQOJQGESWRM2HLMYG6RDZHDEM2I6KZ4FZNOBHAKV24OLLQ, seg_pos={0 0}, upload=439875]: Long tail cancellation due to optimal threshold Dec 26 12:41:51.055384263 (501h12m50.926467308s) [seg_pos={0 0}, upload=439875]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:41:51.055458073 (501h12m50.926541108s) [upload=439875]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:41:51.055464826 (501h12m50.926547871s) [upload=439875]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:41:51.345060146 (501h12m51.216143191s) [upload=439875]: Done uploading segments... Dec 26 12:41:51.345082800 (501h12m51.216165855s) [upload=439875]: Upload finished. err: <nil> Duration: 6.038298387s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XXXX...X.X....X..X...X..XX.....XXXX.....XXX....XX....X..X...] duration: 5.732132584s trace: https://jaeger.dev.storj.io/trace/f97f56868e2aa7c FAIL (2025-12-26 12:43:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: 2AQIKR4J7MCJXU67OL5ANUJ5MKHPDOTDX6GAQWNEORGKBWF7JC3A) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: 4MHNP5IJGVLZNFVEK7VX5CYMX33STTZ5CB736Z3Q4W5JAWKAPL2Q) context deadline exceeded ======LOGS====== Dec 26 12:41:48.442712118 (501h12m48.313795153s) [upload=439875]: Starting upload Dec 26 12:41:48.442786820 (501h12m48.313869885s) [upload=439875]: Done starting upload Dec 26 12:41:48.442961412 (501h12m48.314044447s) [upload=439875]: Uploading segments... Dec 26 12:41:48.443099966 (501h12m48.314183011s) [upload=439875]: Got next segment. Inline: false Dec 26 12:41:48.443286161 (501h12m48.314369206s) [upload=439875]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:41:48.443293645 (501h12m48.31437669s) [upload=439875]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:41:48.691376859 (501h12m48.562459904s) [seg_pos={0 0}, upload=439875]: Begin upload segment... Dec 26 12:41:48.691439108 (501h12m48.562522143s) [seg_pos={0 0}, upload=439875]: Done begin upload segment. Dec 26 12:41:48.691460008 (501h12m48.562543063s) [noise=true , address=45.91.116.8:4048 , piece_id=WM5CKKU336UI5K7JWSIYDS6V36AST7BYYGBNXGMIJYHBRSCTV7CA, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.691489002 (501h12m48.562572047s) [noise=true , address=145.79.227.11:4730 , piece_id=YQKC7KHSNITBZP6VPLES4I6HKQSPQWZQENYTOTF625W2AQPA57SA, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.691613129 (501h12m48.562696164s) [noise=true , address=145.79.227.52:4446 , piece_id=NQSLB7SPF24I3Q3OYY5M3GUHGPLENYQINOBNFP37WVEUPBSVE6TA, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.692179948 (501h12m48.563262993s) [noise=true , address=45.91.116.7:4011 , piece_id=4HYRSFIQOJQGESWRM2HLMYG6RDZHDEM2I6KZ4FZNOBHAKV24OLLQ, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.692193374 (501h12m48.563276419s) [noise=true , address=145.79.227.45:4230 , piece_id=WNEQ4D2ZEPXLIPPD7NFRSCDZJAQREYN2MYUCTVSUI3IWKZLECNFA, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.692668809 (501h12m48.563751854s) [noise=true , address=45.91.116.6:4976 , piece_id=4MHNP5IJGVLZNFVEK7VX5CYMX33STTZ5CB736Z3Q4W5JAWKAPL2Q, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.692706250 (501h12m48.563789285s) [noise=true , address=145.79.227.31:4799 , piece_id=CBATLHCVE6HI77XBXUBVI4AAFQXZJUUEWKMJ6LDLDQ4MHM5VIWSA, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.693084140 (501h12m48.564167185s) [noise=true , address=45.91.116.9:4264 , piece_id=2AQIKR4J7MCJXU67OL5ANUJ5MKHPDOTDX6GAQWNEORGKBWF7JC3A, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.693090893 (501h12m48.564173938s) [noise=true , address=145.79.227.12:4765 , piece_id=2ECTKX35BA2OCYSMI7JRFLGW4HZ5XJNNIDCEUDLSRZPVJLCJSKDA, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.693124386 (501h12m48.564207431s) [noise=true , address=45.91.116.14:4192 , piece_id=H6OJ5UZO7D5PHMBZVLSYI6XTV5GKNGLL2EQNM2WOA36RGHW6CVOA, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.693494481 (501h12m48.564577516s) [noise=true , address=145.79.227.37:4566 , piece_id=EGL5OXQLMH7IBQABZNAUQREEK57CWAP7D2ZAZI7ARXJT2JC5PM5A, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.693930231 (501h12m48.565013276s) [noise=true , address=145.79.227.3:4149 , piece_id=TPOZTDBJEVKZB6SE4DMKMSI7VZQDCO2PZTIPXOYOAMCUCIQTHKQA, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.694324161 (501h12m48.565407197s) [noise=true , address=145.79.227.46:4603 , piece_id=TJZ4R3RBT2ZCB5JCEAF6HGS5D7CFCLBK2JBYFON4XJN7TYTG2NNA, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.694680870 (501h12m48.565763916s) [noise=true , address=145.79.227.41:4086 , piece_id=YAPMPUFKJB4CNXSC6SENO23YGOU5677ZWNSFGZU2FNDZWXFICU7Q, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.694798895 (501h12m48.56588194s) [noise=true , address=145.79.227.36:4517 , piece_id=DE5JLBJWHIHXAFSGKELLYALASXE2PUUFY6VDE6TJKAQGM2UPYZTQ, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.695062958 (501h12m48.566145993s) [noise=true , address=145.79.227.32:4834 , piece_id=3MU6IJPI7BFU7WKCV2JNOMAINNABUMBKNYEBGSCJBBMTM7YXJK6A, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:50.410149874 (501h12m50.281232919s) [upload=439875]: Next returned nil segment Dec 26 12:41:50.410165523 (501h12m50.281248568s) [upload=439875]: Waiting for error group managing segments... Dec 26 12:41:50.690475803 (501h12m50.561558838s) [noise=true , address=145.79.227.11:4730 , piece_id=YQKC7KHSNITBZP6VPLES4I6HKQSPQWZQENYTOTF625W2AQPA57SA, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.699134751 (501h12m50.570217796s) [noise=true , address=145.79.227.45:4230 , piece_id=WNEQ4D2ZEPXLIPPD7NFRSCDZJAQREYN2MYUCTVSUI3IWKZLECNFA, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.699496050 (501h12m50.570579095s) [noise=true , address=145.79.227.52:4446 , piece_id=NQSLB7SPF24I3Q3OYY5M3GUHGPLENYQINOBNFP37WVEUPBSVE6TA, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.705921484 (501h12m50.577004519s) [noise=true , address=145.79.227.3:4149 , piece_id=TPOZTDBJEVKZB6SE4DMKMSI7VZQDCO2PZTIPXOYOAMCUCIQTHKQA, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.706349649 (501h12m50.577432684s) [noise=true , address=145.79.227.37:4566 , piece_id=EGL5OXQLMH7IBQABZNAUQREEK57CWAP7D2ZAZI7ARXJT2JC5PM5A, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.717218657 (501h12m50.588301692s) [noise=true , address=145.79.227.31:4799 , piece_id=CBATLHCVE6HI77XBXUBVI4AAFQXZJUUEWKMJ6LDLDQ4MHM5VIWSA, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.717630781 (501h12m50.588713817s) [noise=true , address=145.79.227.32:4834 , piece_id=3MU6IJPI7BFU7WKCV2JNOMAINNABUMBKNYEBGSCJBBMTM7YXJK6A, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.717969365 (501h12m50.589052401s) [noise=true , address=145.79.227.41:4086 , piece_id=YAPMPUFKJB4CNXSC6SENO23YGOU5677ZWNSFGZU2FNDZWXFICU7Q, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.720454037 (501h12m50.591537082s) [noise=true , address=145.79.227.12:4765 , piece_id=2ECTKX35BA2OCYSMI7JRFLGW4HZ5XJNNIDCEUDLSRZPVJLCJSKDA, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.722154334 (501h12m50.593237379s) [noise=true , address=145.79.227.46:4603 , piece_id=TJZ4R3RBT2ZCB5JCEAF6HGS5D7CFCLBK2JBYFON4XJN7TYTG2NNA, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.728157947 (501h12m50.599240982s) [noise=true , address=145.79.227.36:4517 , piece_id=DE5JLBJWHIHXAFSGKELLYALASXE2PUUFY6VDE6TJKAQGM2UPYZTQ, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.940256359 (501h12m50.811339404s) [noise=true , address=45.91.116.6:4976 , piece_id=4MHNP5IJGVLZNFVEK7VX5CYMX33STTZ5CB736Z3Q4W5JAWKAPL2Q, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:51.054849925 (501h12m50.92593296s) [noise=true , address=45.91.116.9:4264 , piece_id=2AQIKR4J7MCJXU67OL5ANUJ5MKHPDOTDX6GAQWNEORGKBWF7JC3A, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:51.054869883 (501h12m50.925952918s) [seg_pos={0 0}, upload=439875]: Segment reached optimal threshold of 13 pieces. Dec 26 12:41:51.055140669 (501h12m50.926223704s) [noise=true , address=45.91.116.8:4048 , piece_id=WM5CKKU336UI5K7JWSIYDS6V36AST7BYYGBNXGMIJYHBRSCTV7CA, seg_pos={0 0}, upload=439875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 26 12:41:51.055148013 (501h12m50.926231048s) [noise=true , address=45.91.116.8:4048 , piece_id=WM5CKKU336UI5K7JWSIYDS6V36AST7BYYGBNXGMIJYHBRSCTV7CA, seg_pos={0 0}, upload=439875]: Long tail cancellation due to optimal threshold Dec 26 12:41:51.055305071 (501h12m50.926388106s) [noise=true , address=45.91.116.14:4192 , piece_id=H6OJ5UZO7D5PHMBZVLSYI6XTV5GKNGLL2EQNM2WOA36RGHW6CVOA, seg_pos={0 0}, upload=439875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 26 12:41:51.055311344 (501h12m50.926394379s) [noise=true , address=45.91.116.14:4192 , piece_id=H6OJ5UZO7D5PHMBZVLSYI6XTV5GKNGLL2EQNM2WOA36RGHW6CVOA, seg_pos={0 0}, upload=439875]: Long tail cancellation due to optimal threshold Dec 26 12:41:51.055332804 (501h12m50.926415839s) [noise=true , address=45.91.116.7:4011 , piece_id=4HYRSFIQOJQGESWRM2HLMYG6RDZHDEM2I6KZ4FZNOBHAKV24OLLQ, seg_pos={0 0}, upload=439875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 12:41:51.055351250 (501h12m50.926434295s) [noise=true , address=45.91.116.7:4011 , piece_id=4HYRSFIQOJQGESWRM2HLMYG6RDZHDEM2I6KZ4FZNOBHAKV24OLLQ, seg_pos={0 0}, upload=439875]: Long tail cancellation due to optimal threshold Dec 26 12:41:51.055384263 (501h12m50.926467308s) [seg_pos={0 0}, upload=439875]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:41:51.055458073 (501h12m50.926541108s) [upload=439875]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:41:51.055464826 (501h12m50.926547871s) [upload=439875]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:41:51.345060146 (501h12m51.216143191s) [upload=439875]: Done uploading segments... Dec 26 12:41:51.345082800 (501h12m51.216165855s) [upload=439875]: Upload finished. err: <nil> Duration: 5.732160597s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [........X..X......X.......XXX...X.....X.......X.XX......XXX.] duration: 3.7464262s trace: https://jaeger.dev.storj.io/trace/3941b76d19abf99a FAIL (2025-12-26 12:43:43 UTC) orbiter warning: context deadline exceeded Duration: 3.746458283s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................................X....................X.XX] duration: 1m0.003401088s trace: https://jaeger.dev.storj.io/trace/3fa0ab3b9c4265d4 FAIL (2025-12-26 12:43:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== [noise=true , address=77.90.31.15:4717 , piece_id=HGJOKPI43QVML5MXWT3BRJ54WL6H4PISHOUWYTS7NEGNLIEESWRA, seg_pos={0 0}, upload=439887]: Uploading piece... Dec 26 12:42:33.674043219 (501h13m33.545126255s) [noise=true , address=145.79.227.21:4486 , piece_id=JHOLBN3QFBSCWDSJYRXY3UQ56YXNARLNYZDORP7QHMUTMBWUHPEA, seg_pos={0 0}, upload=439887]: Uploading piece... Dec 26 12:42:33.674444594 (501h13m33.545527639s) [noise=true , address=145.79.227.44:4196 , piece_id=BOYB2MXJ6DVUW3EWPIMT62VN6M4I3WXIR5SEUF5JXHKAZMOWJ52Q, seg_pos={0 0}, upload=439887]: Uploading piece... Dec 26 12:42:33.675005302 (501h13m33.546088347s) [noise=true , address=45.91.116.13:4167 , piece_id=MYR23YC2OZ3UPECMTF5ZXNBTAJVFKRXL7MWTB733DLQW3RWC3WUQ, seg_pos={0 0}, upload=439887]: Uploading piece... Dec 26 12:42:33.675433727 (501h13m33.546516772s) [noise=true , address=64.49.13.169:4716 , piece_id=TGQIHWOAMXUQYCKWZSPA5HPBZQIP4OC5D2ECLK376ER7ROZ7VEAQ, seg_pos={0 0}, upload=439887]: Uploading piece... Dec 26 12:42:33.675905005 (501h13m33.54698805s) [noise=true , address=45.91.116.11:4132 , piece_id=2REX4GS2MSHE7M677CBSTP2XCNNO7KC2RXU33Q7EOGAOTLDVTMLA, seg_pos={0 0}, upload=439887]: Uploading piece... Dec 26 12:42:33.676368277 (501h13m33.547451312s) [noise=true , address=77.90.31.11:4569 , piece_id=BSI3I7TLQG5TRHGN7UT5IPYP3YBVLKUZNQNLCZ3IR37WLKB7ZWVA, seg_pos={0 0}, upload=439887]: Uploading piece... Dec 26 12:42:33.676760734 (501h13m33.547843769s) [noise=true , address=145.79.227.30:4796 , piece_id=ZD5DNK67RMPE7ZQGJD5FJP5LM2P326VDXZTIR4V7TAJLMWB5C6CA, seg_pos={0 0}, upload=439887]: Uploading piece... Dec 26 12:42:33.678069847 (501h13m33.549152882s) [noise=true , address=77.90.31.19:4856 , piece_id=N4XM5NDN7RFXBSIIPGHDPWBMRWD6ET3JCRTKVN2F5SYBND4BCAOQ, seg_pos={0 0}, upload=439887]: Uploading piece... Dec 26 12:42:33.679923647 (501h13m33.551006682s) [noise=true , address=77.90.31.14:4692 , piece_id=AWPIXOCPNQERGWTO6KWPDO7PM2YV5LBR3AC756FPC56N5A2CZMWA, seg_pos={0 0}, upload=439887]: Uploading piece... Dec 26 12:42:33.682435710 (501h13m33.553518755s) [noise=true , address=145.79.227.52:4458 , piece_id=GX4L53PZP4VTS3BRWIMBZUZ44AHW3ZCN4HTSQZV5IFXRWZYPS4NQ, seg_pos={0 0}, upload=439887]: Uploading piece... Dec 26 12:42:33.688036255 (501h13m33.55911941s) [noise=true , address=145.79.227.25:4652 , piece_id=G7FYRTQQZVDPEQOOO6SCSV2Q2EPNWBYZ44YIHMELQPKT75DA3KUA, seg_pos={0 0}, upload=439887]: Uploading piece... Dec 26 12:42:33.692644811 (501h13m33.563727856s) [noise=true , address=77.90.31.12:4607 , piece_id=5JWMHH6PBNHXGK4ERP2VPSAAMKXPH2VIPIN4YNVB4IQZ5O5BVAHQ, seg_pos={0 0}, upload=439887]: Uploading piece... Dec 26 12:42:33.694673864 (501h13m33.565756929s) [noise=true , address=77.90.31.10:4527 , piece_id=KW3JZ6IPI3LI7KBQQVGE4AJ2ZQ6WZW5AQQ7BWGIKTF6FIVX7HU4A, seg_pos={0 0}, upload=439887]: Uploading piece... Dec 26 12:42:33.697122076 (501h13m33.568205111s) [noise=true , address=77.90.31.16:4768 , piece_id=LGIKWDJAINTCUYMPKYVXV4XBA35RMIKQHUS37YGGNHIZCXM6EW3A, seg_pos={0 0}, upload=439887]: Uploading piece... Dec 26 12:42:33.706355789 (501h13m33.577438834s) [noise=true , address=145.79.227.40:4077 , piece_id=IPFN23C4GLL663IRDR3BYE6HPTFFXRFKBCUQIP3N3O7BIIPIUVCQ, seg_pos={0 0}, upload=439887]: Uploading piece... Dec 26 12:42:34.995575989 (501h13m34.866659034s) [upload=439887]: Next returned nil segment Dec 26 12:42:34.995592350 (501h13m34.866675385s) [upload=439887]: Waiting for error group managing segments... Dec 26 12:42:35.319805529 (501h13m35.190888574s) [noise=true , address=193.149.164.143:4699 , piece_id=LJPXIB3ND7IQ6CRJ4WWCUVVH756HEU65AM7TGTJRQXPOOSEAO6BA, seg_pos={0 0}, upload=439887]: Done uploading piece. err: <nil> Dec 26 12:42:35.562081554 (501h13m35.433164589s) [noise=true , address=145.79.227.23:4565 , piece_id=EVYBUMLS6KBMWL34A7CPBPEVUNCKRFNI5266FFPIXY6DVBE2CXNA, seg_pos={0 0}, upload=439887]: Done uploading piece. err: <nil> Dec 26 12:42:35.565864438 (501h13m35.436947473s) [noise=true , address=145.79.227.25:4652 , piece_id=G7FYRTQQZVDPEQOOO6SCSV2Q2EPNWBYZ44YIHMELQPKT75DA3KUA, seg_pos={0 0}, upload=439887]: Done uploading piece. err: <nil> Dec 26 12:42:35.573735736 (501h13m35.444818781s) [noise=true , address=145.79.227.21:4486 , piece_id=JHOLBN3QFBSCWDSJYRXY3UQ56YXNARLNYZDORP7QHMUTMBWUHPEA, seg_pos={0 0}, upload=439887]: Done uploading piece. err: <nil> Dec 26 12:42:35.585868057 (501h13m35.456951123s) [noise=true , address=145.79.227.52:4458 , piece_id=GX4L53PZP4VTS3BRWIMBZUZ44AHW3ZCN4HTSQZV5IFXRWZYPS4NQ, seg_pos={0 0}, upload=439887]: Done uploading piece. err: <nil> Dec 26 12:42:35.746695622 (501h13m35.617778667s) [noise=true , address=77.90.31.14:4692 , piece_id=AWPIXOCPNQERGWTO6KWPDO7PM2YV5LBR3AC756FPC56N5A2CZMWA, seg_pos={0 0}, upload=439887]: Done uploading piece. err: <nil> Dec 26 12:42:35.781596639 (501h13m35.652679674s) [noise=true , address=45.91.116.13:4167 , piece_id=MYR23YC2OZ3UPECMTF5ZXNBTAJVFKRXL7MWTB733DLQW3RWC3WUQ, seg_pos={0 0}, upload=439887]: Done uploading piece. err: <nil> Dec 26 12:42:35.812156220 (501h13m35.683239255s) [noise=true , address=45.91.116.11:4132 , piece_id=2REX4GS2MSHE7M677CBSTP2XCNNO7KC2RXU33Q7EOGAOTLDVTMLA, seg_pos={0 0}, upload=439887]: Done uploading piece. err: <nil> Dec 26 12:42:36.032836834 (501h13m35.903919879s) [noise=true , address=77.90.31.11:4569 , piece_id=BSI3I7TLQG5TRHGN7UT5IPYP3YBVLKUZNQNLCZ3IR37WLKB7ZWVA, seg_pos={0 0}, upload=439887]: Done uploading piece. err: <nil> Dec 26 12:42:36.091605471 (501h13m35.962688506s) [noise=true , address=77.90.31.19:4856 , piece_id=N4XM5NDN7RFXBSIIPGHDPWBMRWD6ET3JCRTKVN2F5SYBND4BCAOQ, seg_pos={0 0}, upload=439887]: Done uploading piece. err: <nil> Dec 26 12:42:36.109208519 (501h13m35.980291554s) [noise=true , address=77.90.31.15:4717 , piece_id=HGJOKPI43QVML5MXWT3BRJ54WL6H4PISHOUWYTS7NEGNLIEESWRA, seg_pos={0 0}, upload=439887]: Done uploading piece. err: <nil> Dec 26 12:42:36.505685331 (501h13m36.376768376s) [noise=true , address=145.79.227.44:4196 , piece_id=BOYB2MXJ6DVUW3EWPIMT62VN6M4I3WXIR5SEUF5JXHKAZMOWJ52Q, seg_pos={0 0}, upload=439887]: Done uploading piece. err: <nil> Dec 26 12:42:36.517155672 (501h13m36.388238717s) [noise=true , address=145.79.227.30:4796 , piece_id=ZD5DNK67RMPE7ZQGJD5FJP5LM2P326VDXZTIR4V7TAJLMWB5C6CA, seg_pos={0 0}, upload=439887]: Done uploading piece. err: <nil> Dec 26 12:42:36.520258050 (501h13m36.391341085s) [noise=true , address=145.79.227.40:4077 , piece_id=IPFN23C4GLL663IRDR3BYE6HPTFFXRFKBCUQIP3N3O7BIIPIUVCQ, seg_pos={0 0}, upload=439887]: Done uploading piece. err: <nil> Dec 26 12:42:36.520277978 (501h13m36.391361023s) [seg_pos={0 0}, upload=439887]: Segment reached optimal threshold of 14 pieces. Dec 26 12:42:36.520488989 (501h13m36.391572024s) [noise=true , address=77.90.31.12:4607 , piece_id=5JWMHH6PBNHXGK4ERP2VPSAAMKXPH2VIPIN4YNVB4IQZ5O5BVAHQ, seg_pos={0 0}, upload=439887]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UYNy47GZEha8wG9vZNUAws5ogTBjSXLdt7T2bwqcHGsPrKFrm): protocol: expected piece hash; context canceled; EOF Dec 26 12:42:36.520497205 (501h13m36.39158025s) [noise=true , address=77.90.31.12:4607 , piece_id=5JWMHH6PBNHXGK4ERP2VPSAAMKXPH2VIPIN4YNVB4IQZ5O5BVAHQ, seg_pos={0 0}, upload=439887]: Long tail cancellation due to optimal threshold Dec 26 12:42:36.520500813 (501h13m36.391583858s) [noise=true , address=77.90.31.16:4768 , piece_id=LGIKWDJAINTCUYMPKYVXV4XBA35RMIKQHUS37YGGNHIZCXM6EW3A, seg_pos={0 0}, upload=439887]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UchewmtxBrtSZSpVAJFdZmcTsQVQxkjK9re6cgUGqDhyonQeF): protocol: expected piece hash; context canceled; EOF Dec 26 12:42:36.520509399 (501h13m36.391592434s) [noise=true , address=77.90.31.16:4768 , piece_id=LGIKWDJAINTCUYMPKYVXV4XBA35RMIKQHUS37YGGNHIZCXM6EW3A, seg_pos={0 0}, upload=439887]: Long tail cancellation due to optimal threshold Dec 26 12:42:36.520647041 (501h13m36.391730076s) [noise=true , address=77.90.31.10:4527 , piece_id=KW3JZ6IPI3LI7KBQQVGE4AJ2ZQ6WZW5AQQ7BWGIKTF6FIVX7HU4A, seg_pos={0 0}, upload=439887]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jEyiUUU9NCVFMU8frNggxTTYBMkehsqN4GLsFRkrxubdobGY2): protocol: expected piece hash; context canceled; EOF Dec 26 12:42:36.520654805 (501h13m36.39173784s) [noise=true , address=77.90.31.10:4527 , piece_id=KW3JZ6IPI3LI7KBQQVGE4AJ2ZQ6WZW5AQQ7BWGIKTF6FIVX7HU4A, seg_pos={0 0}, upload=439887]: Long tail cancellation due to optimal threshold Dec 26 12:42:36.520667670 (501h13m36.391750705s) [noise=true , address=64.49.13.169:4716 , piece_id=TGQIHWOAMXUQYCKWZSPA5HPBZQIP4OC5D2ECLK376ER7ROZ7VEAQ, seg_pos={0 0}, upload=439887]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:51796->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:51796->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51796->64.49.13.169:4716: use of closed network connection Dec 26 12:42:36.520675505 (501h13m36.39175855s) [noise=true , address=64.49.13.169:4716 , piece_id=TGQIHWOAMXUQYCKWZSPA5HPBZQIP4OC5D2ECLK376ER7ROZ7VEAQ, seg_pos={0 0}, upload=439887]: Long tail cancellation due to optimal threshold Dec 26 12:42:36.520691415 (501h13m36.39177445s) [seg_pos={0 0}, upload=439887]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 12:42:36.520768201 (501h13m36.391851236s) [upload=439887]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:42:36.520773050 (501h13m36.391856085s) [upload=439887]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:42:36.811005794 (501h13m36.682088849s) [upload=439887]: Done uploading segments... Dec 26 12:42:36.811066690 (501h13m36.682149745s) [upload=439887]: Upload finished. err: <nil> Duration: 1m0.003416567s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XXXXX...X.X....X..X...X..XX.....XXXX.....XXX....XX....X..X..] duration: 6.215658491s trace: https://jaeger.dev.storj.io/trace/f97f56868e2aa7c FAIL (2025-12-26 12:43:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: 2AQIKR4J7MCJXU67OL5ANUJ5MKHPDOTDX6GAQWNEORGKBWF7JC3A) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: 4MHNP5IJGVLZNFVEK7VX5CYMX33STTZ5CB736Z3Q4W5JAWKAPL2Q) context deadline exceeded ======LOGS====== Dec 26 12:41:48.442712118 (501h12m48.313795153s) [upload=439875]: Starting upload Dec 26 12:41:48.442786820 (501h12m48.313869885s) [upload=439875]: Done starting upload Dec 26 12:41:48.442961412 (501h12m48.314044447s) [upload=439875]: Uploading segments... Dec 26 12:41:48.443099966 (501h12m48.314183011s) [upload=439875]: Got next segment. Inline: false Dec 26 12:41:48.443286161 (501h12m48.314369206s) [upload=439875]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 12:41:48.443293645 (501h12m48.31437669s) [upload=439875]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 12:41:48.691376859 (501h12m48.562459904s) [seg_pos={0 0}, upload=439875]: Begin upload segment... Dec 26 12:41:48.691439108 (501h12m48.562522143s) [seg_pos={0 0}, upload=439875]: Done begin upload segment. Dec 26 12:41:48.691460008 (501h12m48.562543063s) [noise=true , address=45.91.116.8:4048 , piece_id=WM5CKKU336UI5K7JWSIYDS6V36AST7BYYGBNXGMIJYHBRSCTV7CA, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.691489002 (501h12m48.562572047s) [noise=true , address=145.79.227.11:4730 , piece_id=YQKC7KHSNITBZP6VPLES4I6HKQSPQWZQENYTOTF625W2AQPA57SA, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.691613129 (501h12m48.562696164s) [noise=true , address=145.79.227.52:4446 , piece_id=NQSLB7SPF24I3Q3OYY5M3GUHGPLENYQINOBNFP37WVEUPBSVE6TA, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.692179948 (501h12m48.563262993s) [noise=true , address=45.91.116.7:4011 , piece_id=4HYRSFIQOJQGESWRM2HLMYG6RDZHDEM2I6KZ4FZNOBHAKV24OLLQ, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.692193374 (501h12m48.563276419s) [noise=true , address=145.79.227.45:4230 , piece_id=WNEQ4D2ZEPXLIPPD7NFRSCDZJAQREYN2MYUCTVSUI3IWKZLECNFA, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.692668809 (501h12m48.563751854s) [noise=true , address=45.91.116.6:4976 , piece_id=4MHNP5IJGVLZNFVEK7VX5CYMX33STTZ5CB736Z3Q4W5JAWKAPL2Q, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.692706250 (501h12m48.563789285s) [noise=true , address=145.79.227.31:4799 , piece_id=CBATLHCVE6HI77XBXUBVI4AAFQXZJUUEWKMJ6LDLDQ4MHM5VIWSA, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.693084140 (501h12m48.564167185s) [noise=true , address=45.91.116.9:4264 , piece_id=2AQIKR4J7MCJXU67OL5ANUJ5MKHPDOTDX6GAQWNEORGKBWF7JC3A, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.693090893 (501h12m48.564173938s) [noise=true , address=145.79.227.12:4765 , piece_id=2ECTKX35BA2OCYSMI7JRFLGW4HZ5XJNNIDCEUDLSRZPVJLCJSKDA, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.693124386 (501h12m48.564207431s) [noise=true , address=45.91.116.14:4192 , piece_id=H6OJ5UZO7D5PHMBZVLSYI6XTV5GKNGLL2EQNM2WOA36RGHW6CVOA, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.693494481 (501h12m48.564577516s) [noise=true , address=145.79.227.37:4566 , piece_id=EGL5OXQLMH7IBQABZNAUQREEK57CWAP7D2ZAZI7ARXJT2JC5PM5A, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.693930231 (501h12m48.565013276s) [noise=true , address=145.79.227.3:4149 , piece_id=TPOZTDBJEVKZB6SE4DMKMSI7VZQDCO2PZTIPXOYOAMCUCIQTHKQA, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.694324161 (501h12m48.565407197s) [noise=true , address=145.79.227.46:4603 , piece_id=TJZ4R3RBT2ZCB5JCEAF6HGS5D7CFCLBK2JBYFON4XJN7TYTG2NNA, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.694680870 (501h12m48.565763916s) [noise=true , address=145.79.227.41:4086 , piece_id=YAPMPUFKJB4CNXSC6SENO23YGOU5677ZWNSFGZU2FNDZWXFICU7Q, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.694798895 (501h12m48.56588194s) [noise=true , address=145.79.227.36:4517 , piece_id=DE5JLBJWHIHXAFSGKELLYALASXE2PUUFY6VDE6TJKAQGM2UPYZTQ, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:48.695062958 (501h12m48.566145993s) [noise=true , address=145.79.227.32:4834 , piece_id=3MU6IJPI7BFU7WKCV2JNOMAINNABUMBKNYEBGSCJBBMTM7YXJK6A, seg_pos={0 0}, upload=439875]: Uploading piece... Dec 26 12:41:50.410149874 (501h12m50.281232919s) [upload=439875]: Next returned nil segment Dec 26 12:41:50.410165523 (501h12m50.281248568s) [upload=439875]: Waiting for error group managing segments... Dec 26 12:41:50.690475803 (501h12m50.561558838s) [noise=true , address=145.79.227.11:4730 , piece_id=YQKC7KHSNITBZP6VPLES4I6HKQSPQWZQENYTOTF625W2AQPA57SA, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.699134751 (501h12m50.570217796s) [noise=true , address=145.79.227.45:4230 , piece_id=WNEQ4D2ZEPXLIPPD7NFRSCDZJAQREYN2MYUCTVSUI3IWKZLECNFA, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.699496050 (501h12m50.570579095s) [noise=true , address=145.79.227.52:4446 , piece_id=NQSLB7SPF24I3Q3OYY5M3GUHGPLENYQINOBNFP37WVEUPBSVE6TA, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.705921484 (501h12m50.577004519s) [noise=true , address=145.79.227.3:4149 , piece_id=TPOZTDBJEVKZB6SE4DMKMSI7VZQDCO2PZTIPXOYOAMCUCIQTHKQA, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.706349649 (501h12m50.577432684s) [noise=true , address=145.79.227.37:4566 , piece_id=EGL5OXQLMH7IBQABZNAUQREEK57CWAP7D2ZAZI7ARXJT2JC5PM5A, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.717218657 (501h12m50.588301692s) [noise=true , address=145.79.227.31:4799 , piece_id=CBATLHCVE6HI77XBXUBVI4AAFQXZJUUEWKMJ6LDLDQ4MHM5VIWSA, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.717630781 (501h12m50.588713817s) [noise=true , address=145.79.227.32:4834 , piece_id=3MU6IJPI7BFU7WKCV2JNOMAINNABUMBKNYEBGSCJBBMTM7YXJK6A, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.717969365 (501h12m50.589052401s) [noise=true , address=145.79.227.41:4086 , piece_id=YAPMPUFKJB4CNXSC6SENO23YGOU5677ZWNSFGZU2FNDZWXFICU7Q, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.720454037 (501h12m50.591537082s) [noise=true , address=145.79.227.12:4765 , piece_id=2ECTKX35BA2OCYSMI7JRFLGW4HZ5XJNNIDCEUDLSRZPVJLCJSKDA, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.722154334 (501h12m50.593237379s) [noise=true , address=145.79.227.46:4603 , piece_id=TJZ4R3RBT2ZCB5JCEAF6HGS5D7CFCLBK2JBYFON4XJN7TYTG2NNA, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.728157947 (501h12m50.599240982s) [noise=true , address=145.79.227.36:4517 , piece_id=DE5JLBJWHIHXAFSGKELLYALASXE2PUUFY6VDE6TJKAQGM2UPYZTQ, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:50.940256359 (501h12m50.811339404s) [noise=true , address=45.91.116.6:4976 , piece_id=4MHNP5IJGVLZNFVEK7VX5CYMX33STTZ5CB736Z3Q4W5JAWKAPL2Q, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:51.054849925 (501h12m50.92593296s) [noise=true , address=45.91.116.9:4264 , piece_id=2AQIKR4J7MCJXU67OL5ANUJ5MKHPDOTDX6GAQWNEORGKBWF7JC3A, seg_pos={0 0}, upload=439875]: Done uploading piece. err: <nil> Dec 26 12:41:51.054869883 (501h12m50.925952918s) [seg_pos={0 0}, upload=439875]: Segment reached optimal threshold of 13 pieces. Dec 26 12:41:51.055140669 (501h12m50.926223704s) [noise=true , address=45.91.116.8:4048 , piece_id=WM5CKKU336UI5K7JWSIYDS6V36AST7BYYGBNXGMIJYHBRSCTV7CA, seg_pos={0 0}, upload=439875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 26 12:41:51.055148013 (501h12m50.926231048s) [noise=true , address=45.91.116.8:4048 , piece_id=WM5CKKU336UI5K7JWSIYDS6V36AST7BYYGBNXGMIJYHBRSCTV7CA, seg_pos={0 0}, upload=439875]: Long tail cancellation due to optimal threshold Dec 26 12:41:51.055305071 (501h12m50.926388106s) [noise=true , address=45.91.116.14:4192 , piece_id=H6OJ5UZO7D5PHMBZVLSYI6XTV5GKNGLL2EQNM2WOA36RGHW6CVOA, seg_pos={0 0}, upload=439875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 26 12:41:51.055311344 (501h12m50.926394379s) [noise=true , address=45.91.116.14:4192 , piece_id=H6OJ5UZO7D5PHMBZVLSYI6XTV5GKNGLL2EQNM2WOA36RGHW6CVOA, seg_pos={0 0}, upload=439875]: Long tail cancellation due to optimal threshold Dec 26 12:41:51.055332804 (501h12m50.926415839s) [noise=true , address=45.91.116.7:4011 , piece_id=4HYRSFIQOJQGESWRM2HLMYG6RDZHDEM2I6KZ4FZNOBHAKV24OLLQ, seg_pos={0 0}, upload=439875]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 26 12:41:51.055351250 (501h12m50.926434295s) [noise=true , address=45.91.116.7:4011 , piece_id=4HYRSFIQOJQGESWRM2HLMYG6RDZHDEM2I6KZ4FZNOBHAKV24OLLQ, seg_pos={0 0}, upload=439875]: Long tail cancellation due to optimal threshold Dec 26 12:41:51.055384263 (501h12m50.926467308s) [seg_pos={0 0}, upload=439875]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 12:41:51.055458073 (501h12m50.926541108s) [upload=439875]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:41:51.055464826 (501h12m50.926547871s) [upload=439875]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:41:51.345060146 (501h12m51.216143191s) [upload=439875]: Done uploading segments... Dec 26 12:41:51.345082800 (501h12m51.216165855s) [upload=439875]: Upload finished. err: <nil> Duration: 6.215688929s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [X........X..X......X.......XXX...X.....X.......X.XX......XXX] duration: 5.001547813s trace: https://jaeger.dev.storj.io/trace/3941b76d19abf99a FAIL (2025-12-26 12:43:13 UTC) orbiter warning: context deadline exceeded Duration: 5.001565177s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................X....................X.X] duration: 1m0.003436126s trace: https://jaeger.dev.storj.io/trace/528b991f18e6bb9 FAIL (2025-12-26 12:42:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, Piece ID: S3KS6PKKA5Y5IOO6DC2FDYNINUUMT65YNUWUHNG4S4BPHLNBRSZQ) context deadline exceeded; piecestore: (Node ID: 12QnZ3q9oe4zPuLpS3V1DCBhyaU1umS5B7bpuAqPkaaY96ErWbo, Piece ID: AMC2B5AN6CKKRL4ILPVIOJCYEB65TN732VXXVUHJI7CVRA2PDUNQ) context deadline exceeded; piecestore: (Node ID: 19DfbHig9tzF4VRHtSWkExwUS2bdZ8GFgKnAD9wvMBG7qaesNa, Piece ID: 3R3P5AZORVFXV7WFSQVJZBXM5G4BIKZOY5TKCOZAY6I7NYL4FIJA) context deadline exceeded ======LOGS====== eg_pos={0 0}, upload=439872]: Uploading piece... Dec 26 12:41:33.428402134 (501h12m33.299485169s) [noise=true , address=77.90.31.18:4832 , piece_id=AMC2B5AN6CKKRL4ILPVIOJCYEB65TN732VXXVUHJI7CVRA2PDUNQ, seg_pos={0 0}, upload=439872]: Uploading piece... Dec 26 12:41:33.428827153 (501h12m33.299910188s) [noise=true , address=77.90.31.11:4574 , piece_id=S3KS6PKKA5Y5IOO6DC2FDYNINUUMT65YNUWUHNG4S4BPHLNBRSZQ, seg_pos={0 0}, upload=439872]: Uploading piece... Dec 26 12:41:33.429139288 (501h12m33.300222323s) [noise=true , address=77.90.31.15:4703 , piece_id=DAGS3MST2GTGDJ2AZDN54KUEGKWMC5NQTLAZCSY5KWNBTJCYUJ3A, seg_pos={0 0}, upload=439872]: Uploading piece... Dec 26 12:41:33.429468875 (501h12m33.30055191s) [noise=true , address=77.90.31.19:4847 , piece_id=KDSXBVF636LWOH5O5DE3CQX3NSBW4H62UM7FJWU65WXLSX5TKYPA, seg_pos={0 0}, upload=439872]: Uploading piece... Dec 26 12:41:33.430303073 (501h12m33.301386108s) [noise=true , address=45.91.116.7:4043 , piece_id=Y4QHRI43DKD73VIMHSHQX4V7B54FIWLUD5UT26VMMLO2MDWUQGQA, seg_pos={0 0}, upload=439872]: Uploading piece... Dec 26 12:41:33.430773760 (501h12m33.301856795s) [noise=true , address=64.49.13.169:4711 , piece_id=6B5NYPFTYU3SV6BRFKILQ4SOXG6UAK6Z67NTGXB64VN4HPEYJFYA, seg_pos={0 0}, upload=439872]: Uploading piece... Dec 26 12:41:33.431148644 (501h12m33.302231679s) [noise=true , address=145.79.227.34:4920 , piece_id=44PBAF3YSJMGG47FDC5WKQMYQCU2TWMMAYBTM64HN2UZX5ZWPRZA, seg_pos={0 0}, upload=439872]: Uploading piece... Dec 26 12:41:33.431544037 (501h12m33.302627072s) [noise=true , address=145.79.227.44:4206 , piece_id=D5DQPLJLO4DXR4NUO2EHX4WBMUXUNLJV6SFZNBGQ7UEBGIQCUKBA, seg_pos={0 0}, upload=439872]: Uploading piece... Dec 26 12:41:33.432026586 (501h12m33.303109621s) [noise=true , address=145.79.227.46:4614 , piece_id=PZTAPGGCUMCIVMCU62BSNPDCVKON6H3I6S43VEJI24DB36ZXTIAQ, seg_pos={0 0}, upload=439872]: Uploading piece... Dec 26 12:41:33.432161653 (501h12m33.303244688s) [noise=true , address=145.79.227.30:4764 , piece_id=F7HNTXGKDSFMUTHT2CDYJPSBRHC2JDMXVQ7H37QO5OIDTQVA3CTA, seg_pos={0 0}, upload=439872]: Uploading piece... Dec 26 12:41:33.432868038 (501h12m33.303951073s) [noise=true , address=77.90.31.14:4693 , piece_id=UCMLUU6K5GOCNXXUETIZ63F2DK67Q7KXRI465HAYKPNFTZX6G3RA, seg_pos={0 0}, upload=439872]: Uploading piece... Dec 26 12:41:33.432939674 (501h12m33.304022709s) [noise=true , address=145.79.227.40:4080 , piece_id=KFTCTPLPDLNPSUIFXK3UKB36L72PLHIAAFMDP5X2BYFCPF7HOMIA, seg_pos={0 0}, upload=439872]: Uploading piece... Dec 26 12:41:34.885056619 (501h12m34.756139654s) [upload=439872]: Next returned nil segment Dec 26 12:41:34.885067610 (501h12m34.756150645s) [upload=439872]: Waiting for error group managing segments... Dec 26 12:41:35.531529226 (501h12m35.402612271s) [noise=true , address=77.90.31.18:4832 , piece_id=AMC2B5AN6CKKRL4ILPVIOJCYEB65TN732VXXVUHJI7CVRA2PDUNQ, seg_pos={0 0}, upload=439872]: Done uploading piece. err: <nil> Dec 26 12:41:35.592313670 (501h12m35.463396705s) [noise=true , address=145.79.227.46:4614 , piece_id=PZTAPGGCUMCIVMCU62BSNPDCVKON6H3I6S43VEJI24DB36ZXTIAQ, seg_pos={0 0}, upload=439872]: Done uploading piece. err: <nil> Dec 26 12:41:35.594465949 (501h12m35.465548984s) [noise=true , address=77.90.31.15:4703 , piece_id=DAGS3MST2GTGDJ2AZDN54KUEGKWMC5NQTLAZCSY5KWNBTJCYUJ3A, seg_pos={0 0}, upload=439872]: Done uploading piece. err: <nil> Dec 26 12:41:35.630650019 (501h12m35.501733054s) [noise=true , address=45.91.116.9:4286 , piece_id=3R3P5AZORVFXV7WFSQVJZBXM5G4BIKZOY5TKCOZAY6I7NYL4FIJA, seg_pos={0 0}, upload=439872]: Done uploading piece. err: <nil> Dec 26 12:41:35.930632729 (501h12m35.801715774s) [noise=true , address=45.91.116.4:4949 , piece_id=RQMHHYCPO6CXJ462Z725N2UCFGL6BDZUVSPYJ3WJE7RDUW66PPBQ, seg_pos={0 0}, upload=439872]: Done uploading piece. err: <nil> Dec 26 12:41:36.002023399 (501h12m35.873106435s) [noise=true , address=77.90.31.13:4645 , piece_id=7MIL2HBR24PFRL2KTVTGVP72DYWTRGOGBMPXAC4TYKG7UFOE24TQ, seg_pos={0 0}, upload=439872]: Done uploading piece. err: <nil> Dec 26 12:41:36.174133096 (501h12m36.045216141s) [noise=true , address=145.79.227.44:4206 , piece_id=D5DQPLJLO4DXR4NUO2EHX4WBMUXUNLJV6SFZNBGQ7UEBGIQCUKBA, seg_pos={0 0}, upload=439872]: Done uploading piece. err: <nil> Dec 26 12:41:36.200651902 (501h12m36.071734938s) [noise=true , address=145.79.227.40:4080 , piece_id=KFTCTPLPDLNPSUIFXK3UKB36L72PLHIAAFMDP5X2BYFCPF7HOMIA, seg_pos={0 0}, upload=439872]: Done uploading piece. err: <nil> Dec 26 12:41:36.212589173 (501h12m36.083672218s) [noise=true , address=145.79.227.30:4764 , piece_id=F7HNTXGKDSFMUTHT2CDYJPSBRHC2JDMXVQ7H37QO5OIDTQVA3CTA, seg_pos={0 0}, upload=439872]: Done uploading piece. err: <nil> Dec 26 12:41:36.271113744 (501h12m36.142196789s) [noise=true , address=45.91.116.7:4043 , piece_id=Y4QHRI43DKD73VIMHSHQX4V7B54FIWLUD5UT26VMMLO2MDWUQGQA, seg_pos={0 0}, upload=439872]: Done uploading piece. err: <nil> Dec 26 12:41:36.440774838 (501h12m36.311857903s) [noise=true , address=77.90.31.11:4574 , piece_id=S3KS6PKKA5Y5IOO6DC2FDYNINUUMT65YNUWUHNG4S4BPHLNBRSZQ, seg_pos={0 0}, upload=439872]: Done uploading piece. err: <nil> Dec 26 12:41:36.522445654 (501h12m36.393528709s) [noise=true , address=77.90.31.19:4847 , piece_id=KDSXBVF636LWOH5O5DE3CQX3NSBW4H62UM7FJWU65WXLSX5TKYPA, seg_pos={0 0}, upload=439872]: Done uploading piece. err: <nil> Dec 26 12:41:36.643559776 (501h12m36.514642811s) [noise=true , address=193.149.164.143:4700 , piece_id=ZD23WAV2RT33PJN4ZEAWBAYPOND5SLD4ZX7X47Z4EB7XXHJQX2CA, seg_pos={0 0}, upload=439872]: Done uploading piece. err: <nil> Dec 26 12:41:36.660088801 (501h12m36.531171836s) [noise=true , address=145.79.227.34:4920 , piece_id=44PBAF3YSJMGG47FDC5WKQMYQCU2TWMMAYBTM64HN2UZX5ZWPRZA, seg_pos={0 0}, upload=439872]: Done uploading piece. err: <nil> Dec 26 12:41:36.660113177 (501h12m36.531196222s) [seg_pos={0 0}, upload=439872]: Segment reached optimal threshold of 14 pieces. Dec 26 12:41:36.660374584 (501h12m36.53145763s) [noise=true , address=64.49.13.169:4706 , piece_id=WH6KT4P2JQNCPBMGCF5CLHN6ZICR2UIEK63SPSSZGZSKSDCFW7XA, seg_pos={0 0}, upload=439872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:51992->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:51992->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51992->64.49.13.169:4706: use of closed network connection Dec 26 12:41:36.660384244 (501h12m36.531467299s) [noise=true , address=64.49.13.169:4706 , piece_id=WH6KT4P2JQNCPBMGCF5CLHN6ZICR2UIEK63SPSSZGZSKSDCFW7XA, seg_pos={0 0}, upload=439872]: Long tail cancellation due to optimal threshold Dec 26 12:41:36.660696939 (501h12m36.531779974s) [noise=true , address=64.49.13.169:4711 , piece_id=6B5NYPFTYU3SV6BRFKILQ4SOXG6UAK6Z67NTGXB64VN4HPEYJFYA, seg_pos={0 0}, upload=439872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:46052->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:46052->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46052->64.49.13.169:4711: use of closed network connection Dec 26 12:41:36.660704804 (501h12m36.531787849s) [noise=true , address=64.49.13.169:4711 , piece_id=6B5NYPFTYU3SV6BRFKILQ4SOXG6UAK6Z67NTGXB64VN4HPEYJFYA, seg_pos={0 0}, upload=439872]: Long tail cancellation due to optimal threshold Dec 26 12:41:36.660839410 (501h12m36.531922465s) [noise=true , address=77.90.31.14:4693 , piece_id=UCMLUU6K5GOCNXXUETIZ63F2DK67Q7KXRI465HAYKPNFTZX6G3RA, seg_pos={0 0}, upload=439872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey): protocol: write tcp 188.166.225.170:47418->77.90.31.14:4693: use of closed network connection; write tcp 188.166.225.170:47418->77.90.31.14:4693: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47418->77.90.31.14:4693: use of closed network connection Dec 26 12:41:36.660846173 (501h12m36.531929228s) [noise=true , address=77.90.31.14:4693 , piece_id=UCMLUU6K5GOCNXXUETIZ63F2DK67Q7KXRI465HAYKPNFTZX6G3RA, seg_pos={0 0}, upload=439872]: Long tail cancellation due to optimal threshold Dec 26 12:41:36.660964798 (501h12m36.532047843s) [noise=true , address=64.49.13.169:4716 , piece_id=M3USONSMDWBCUMTF4EBMHSA6BMLKOVD7CBXSKVU6WDDGKAMMHPDQ, seg_pos={0 0}, upload=439872]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:49712->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:49712->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49712->64.49.13.169:4716: use of closed network connection Dec 26 12:41:36.660971922 (501h12m36.532054967s) [noise=true , address=64.49.13.169:4716 , piece_id=M3USONSMDWBCUMTF4EBMHSA6BMLKOVD7CBXSKVU6WDDGKAMMHPDQ, seg_pos={0 0}, upload=439872]: Long tail cancellation due to optimal threshold Dec 26 12:41:36.660990918 (501h12m36.532073953s) [seg_pos={0 0}, upload=439872]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 12:41:36.661071662 (501h12m36.532154707s) [upload=439872]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 12:41:36.661078274 (501h12m36.532161309s) [upload=439872]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 12:41:36.957367101 (501h12m36.828450146s) [upload=439872]: Done uploading segments... Dec 26 12:41:36.957386658 (501h12m36.828469703s) [upload=439872]: Upload finished. err: <nil> Duration: 1m0.003450323s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)