Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [...X.X.XX..XX.....XX.X.....X.X..XXXX..XXX.XX...X.X.....XXX..] duration: 5.762727589s trace: https://jaeger.dev.storj.io/trace/70603e9358c10f6c FAIL (2025-12-29 03:33:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: OZLPMBLJICM5NOPAEBABB3YXYUGUJUB5IKGY2OE7GNNWE5UQA4XQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: OWG2ZNE3UPFRYNUEDBW2D3732OO6ZSNNZ3ILHJUZUPXAQSN2RFMA) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: LVXDUSPX73DWQHEK4Z5WH7NYQTDYFTAM2ALLVCHNDCLHECQ7PWKQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: BXPX4L6XFGAHAMZ6G6Q27D7VJOOBGIBPWUOJXC3EFN6RXCLL2VVA) context deadline exceeded ======LOGS====== 4 (564h2m40.033183739s) [upload=494534]: Uploading segments... Dec 29 03:31:40.162237635 (564h2m40.03332067s) [upload=494534]: Got next segment. Inline: false Dec 29 03:31:40.162275738 (564h2m40.033358773s) [upload=494534]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:31:40.162280206 (564h2m40.033363241s) [upload=494534]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:31:40.403193059 (564h2m40.274276094s) [seg_pos={0 0}, upload=494534]: Begin upload segment... Dec 29 03:31:40.403243144 (564h2m40.274326179s) [seg_pos={0 0}, upload=494534]: Done begin upload segment. Dec 29 03:31:40.403269044 (564h2m40.274352079s) [noise=true , address=77.90.31.10:4520 , piece_id=AEG4JRHDZISDIA776YW32SFHVP2B5MHL3N2VAA2TVJK5N26XUIKA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.407207904 (564h2m40.278290939s) [noise=true , address=45.91.116.6:4973 , piece_id=BXPX4L6XFGAHAMZ6G6Q27D7VJOOBGIBPWUOJXC3EFN6RXCLL2VVA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.408387570 (564h2m40.279470605s) [noise=true , address=145.79.227.18:4006 , piece_id=BWFQ24R7OSSOA3XHBDI24VJDYJOARW6JVRI5OAFR7IKNX4TVFWFA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.409424213 (564h2m40.280507248s) [noise=true , address=45.91.116.14:4189 , piece_id=OZLPMBLJICM5NOPAEBABB3YXYUGUJUB5IKGY2OE7GNNWE5UQA4XQ, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.410016982 (564h2m40.281100017s) [noise=true , address=77.90.31.13:4629 , piece_id=OWG2ZNE3UPFRYNUEDBW2D3732OO6ZSNNZ3ILHJUZUPXAQSN2RFMA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.411153306 (564h2m40.282236351s) [noise=true , address=77.90.31.12:4589 , piece_id=LVXDUSPX73DWQHEK4Z5WH7NYQTDYFTAM2ALLVCHNDCLHECQ7PWKQ, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.412201522 (564h2m40.283284557s) [noise=true , address=145.79.227.6:4259 , piece_id=ASTLNUGO5PX5DB5ALAQ3SCXIGMTRJIQCWNG5WFXA5OQN4HL2ZTXA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.413316305 (564h2m40.28439935s) [noise=true , address=145.79.227.11:4730 , piece_id=I6XK6IBGRE6RTB67BS6WYQG3YCYLSNSKS2CKIWGBU4YPTN2ZRY2Q, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.414450344 (564h2m40.28553338s) [noise=true , address=45.91.116.12:4228 , piece_id=LTJD3KG2O6GST5OVD4KI7HFAOOO5HJXRRETMBSEOUGN2CMOG5CTA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.415577741 (564h2m40.286660776s) [noise=true , address=145.79.227.32:4834 , piece_id=2YGMRVRYGLKI6ZMTFOZCX27XPXV3ZYB5RPOWZE46NIQFDRFQZ4CA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.416682445 (564h2m40.28776548s) [noise=true , address=45.91.116.11:4118 , piece_id=HUYHCJPUL3GAL3TF3H4TZ65OS6YWNDGND4K6KYKJWKMIKREKXVHA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.417003255 (564h2m40.28808629s) [noise=true , address=145.79.227.43:4158 , piece_id=VCDQ6327BCGUDSH56LO6D6KE7ZQUTOPX2J4PCTY3NCCGDVC2DZOQ, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.418148246 (564h2m40.289231321s) [noise=true , address=145.79.227.47:4267 , piece_id=XJP43VGLYA3UZHVCZNAQU36KH6VWVIJ5A7SG54JL7UTDPS6XBDYA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.418174656 (564h2m40.289257691s) [noise=true , address=45.91.116.5:4300 , piece_id=HL2XWR7XYFWW7K5H634EQ3EZHP2PEMVMVVMDKJ75TJ3XBT4RQGKA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.419339374 (564h2m40.290422409s) [noise=true , address=45.91.116.9:4264 , piece_id=N6OBFWQGNCFXHJ2RWVEZROZOOCPLBS4NTQAQNO3I5KW7I4RLFL6Q, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.420371149 (564h2m40.291454184s) [noise=true , address=145.79.227.37:4565 , piece_id=WJWLDBH2U2QVP6ELF4Y2PCIBDKTMGHU7W4MY3PTXB3YKIADEJKNA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:42.017668772 (564h2m41.888751807s) [upload=494534]: Next returned nil segment Dec 29 03:31:42.017681617 (564h2m41.888764662s) [upload=494534]: Waiting for error group managing segments... Dec 29 03:31:42.391922515 (564h2m42.26300556s) [noise=true , address=145.79.227.18:4006 , piece_id=BWFQ24R7OSSOA3XHBDI24VJDYJOARW6JVRI5OAFR7IKNX4TVFWFA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.408585584 (564h2m42.279668619s) [noise=true , address=145.79.227.32:4834 , piece_id=2YGMRVRYGLKI6ZMTFOZCX27XPXV3ZYB5RPOWZE46NIQFDRFQZ4CA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.412644553 (564h2m42.283727598s) [noise=true , address=145.79.227.37:4565 , piece_id=WJWLDBH2U2QVP6ELF4Y2PCIBDKTMGHU7W4MY3PTXB3YKIADEJKNA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.425607085 (564h2m42.29669012s) [noise=true , address=145.79.227.47:4267 , piece_id=XJP43VGLYA3UZHVCZNAQU36KH6VWVIJ5A7SG54JL7UTDPS6XBDYA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.431056222 (564h2m42.302139257s) [noise=true , address=145.79.227.11:4730 , piece_id=I6XK6IBGRE6RTB67BS6WYQG3YCYLSNSKS2CKIWGBU4YPTN2ZRY2Q, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.590366150 (564h2m42.461449185s) [noise=true , address=145.79.227.6:4259 , piece_id=ASTLNUGO5PX5DB5ALAQ3SCXIGMTRJIQCWNG5WFXA5OQN4HL2ZTXA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.643846905 (564h2m42.51492995s) [noise=true , address=45.91.116.9:4264 , piece_id=N6OBFWQGNCFXHJ2RWVEZROZOOCPLBS4NTQAQNO3I5KW7I4RLFL6Q, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.650832878 (564h2m42.521915933s) [noise=true , address=77.90.31.12:4589 , piece_id=LVXDUSPX73DWQHEK4Z5WH7NYQTDYFTAM2ALLVCHNDCLHECQ7PWKQ, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.661378509 (564h2m42.532461544s) [noise=true , address=45.91.116.14:4189 , piece_id=OZLPMBLJICM5NOPAEBABB3YXYUGUJUB5IKGY2OE7GNNWE5UQA4XQ, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.666198777 (564h2m42.537281812s) [noise=true , address=77.90.31.10:4520 , piece_id=AEG4JRHDZISDIA776YW32SFHVP2B5MHL3N2VAA2TVJK5N26XUIKA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.995773675 (564h2m42.86685672s) [noise=true , address=77.90.31.13:4629 , piece_id=OWG2ZNE3UPFRYNUEDBW2D3732OO6ZSNNZ3ILHJUZUPXAQSN2RFMA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:43.030874201 (564h2m42.901957236s) [noise=true , address=45.91.116.11:4118 , piece_id=HUYHCJPUL3GAL3TF3H4TZ65OS6YWNDGND4K6KYKJWKMIKREKXVHA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:43.403788173 (564h2m43.274871218s) [noise=true , address=45.91.116.6:4973 , piece_id=BXPX4L6XFGAHAMZ6G6Q27D7VJOOBGIBPWUOJXC3EFN6RXCLL2VVA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:43.403812419 (564h2m43.274895464s) [seg_pos={0 0}, upload=494534]: Segment reached optimal threshold of 13 pieces. Dec 29 03:31:43.404082474 (564h2m43.275165549s) [noise=true , address=145.79.227.43:4158 , piece_id=VCDQ6327BCGUDSH56LO6D6KE7ZQUTOPX2J4PCTY3NCCGDVC2DZOQ, seg_pos={0 0}, upload=494534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 29 03:31:43.404094617 (564h2m43.275177662s) [noise=true , address=145.79.227.43:4158 , piece_id=VCDQ6327BCGUDSH56LO6D6KE7ZQUTOPX2J4PCTY3NCCGDVC2DZOQ, seg_pos={0 0}, upload=494534]: Long tail cancellation due to optimal threshold Dec 29 03:31:43.404310327 (564h2m43.275393362s) [noise=true , address=45.91.116.12:4228 , piece_id=LTJD3KG2O6GST5OVD4KI7HFAOOO5HJXRRETMBSEOUGN2CMOG5CTA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:38814->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:38814->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38814->45.91.116.12:4228: use of closed network connection Dec 29 03:31:43.404339703 (564h2m43.275422748s) [noise=true , address=45.91.116.12:4228 , piece_id=LTJD3KG2O6GST5OVD4KI7HFAOOO5HJXRRETMBSEOUGN2CMOG5CTA, seg_pos={0 0}, upload=494534]: Long tail cancellation due to optimal threshold Dec 29 03:31:43.404392834 (564h2m43.275475869s) [noise=true , address=45.91.116.5:4300 , piece_id=HL2XWR7XYFWW7K5H634EQ3EZHP2PEMVMVVMDKJ75TJ3XBT4RQGKA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:49458->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:49458->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49458->45.91.116.5:4300: use of closed network connection Dec 29 03:31:43.404401231 (564h2m43.275484266s) [noise=true , address=45.91.116.5:4300 , piece_id=HL2XWR7XYFWW7K5H634EQ3EZHP2PEMVMVVMDKJ75TJ3XBT4RQGKA, seg_pos={0 0}, upload=494534]: Long tail cancellation due to optimal threshold Dec 29 03:31:43.404420036 (564h2m43.275503071s) [seg_pos={0 0}, upload=494534]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:31:43.404491923 (564h2m43.275574978s) [upload=494534]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:31:43.404497894 (564h2m43.27558095s) [upload=494534]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:31:43.695226022 (564h2m43.566309057s) [upload=494534]: Done uploading segments... Dec 29 03:31:43.695245209 (564h2m43.566328254s) [upload=494534]: Upload finished. err: <nil> Duration: 5.762773776s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................................XX................] duration: 3.456734776s trace: https://jaeger.dev.storj.io/trace/481fdf974580574a FAIL (2025-12-29 03:33:09 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.45676216s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........................................XX...............] duration: 3.48200628s trace: https://jaeger.dev.storj.io/trace/481fdf974580574a FAIL (2025-12-29 03:32:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.482022762s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [....X.X.XX..XX.....XX.X.....X.X..XXXX..XXX.XX...X.X.....XXX.] duration: 6.019191115s trace: https://jaeger.dev.storj.io/trace/70603e9358c10f6c FAIL (2025-12-29 03:32:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: OZLPMBLJICM5NOPAEBABB3YXYUGUJUB5IKGY2OE7GNNWE5UQA4XQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: OWG2ZNE3UPFRYNUEDBW2D3732OO6ZSNNZ3ILHJUZUPXAQSN2RFMA) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: LVXDUSPX73DWQHEK4Z5WH7NYQTDYFTAM2ALLVCHNDCLHECQ7PWKQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: BXPX4L6XFGAHAMZ6G6Q27D7VJOOBGIBPWUOJXC3EFN6RXCLL2VVA) context deadline exceeded ======LOGS====== 4 (564h2m40.033183739s) [upload=494534]: Uploading segments... Dec 29 03:31:40.162237635 (564h2m40.03332067s) [upload=494534]: Got next segment. Inline: false Dec 29 03:31:40.162275738 (564h2m40.033358773s) [upload=494534]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:31:40.162280206 (564h2m40.033363241s) [upload=494534]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:31:40.403193059 (564h2m40.274276094s) [seg_pos={0 0}, upload=494534]: Begin upload segment... Dec 29 03:31:40.403243144 (564h2m40.274326179s) [seg_pos={0 0}, upload=494534]: Done begin upload segment. Dec 29 03:31:40.403269044 (564h2m40.274352079s) [noise=true , address=77.90.31.10:4520 , piece_id=AEG4JRHDZISDIA776YW32SFHVP2B5MHL3N2VAA2TVJK5N26XUIKA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.407207904 (564h2m40.278290939s) [noise=true , address=45.91.116.6:4973 , piece_id=BXPX4L6XFGAHAMZ6G6Q27D7VJOOBGIBPWUOJXC3EFN6RXCLL2VVA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.408387570 (564h2m40.279470605s) [noise=true , address=145.79.227.18:4006 , piece_id=BWFQ24R7OSSOA3XHBDI24VJDYJOARW6JVRI5OAFR7IKNX4TVFWFA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.409424213 (564h2m40.280507248s) [noise=true , address=45.91.116.14:4189 , piece_id=OZLPMBLJICM5NOPAEBABB3YXYUGUJUB5IKGY2OE7GNNWE5UQA4XQ, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.410016982 (564h2m40.281100017s) [noise=true , address=77.90.31.13:4629 , piece_id=OWG2ZNE3UPFRYNUEDBW2D3732OO6ZSNNZ3ILHJUZUPXAQSN2RFMA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.411153306 (564h2m40.282236351s) [noise=true , address=77.90.31.12:4589 , piece_id=LVXDUSPX73DWQHEK4Z5WH7NYQTDYFTAM2ALLVCHNDCLHECQ7PWKQ, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.412201522 (564h2m40.283284557s) [noise=true , address=145.79.227.6:4259 , piece_id=ASTLNUGO5PX5DB5ALAQ3SCXIGMTRJIQCWNG5WFXA5OQN4HL2ZTXA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.413316305 (564h2m40.28439935s) [noise=true , address=145.79.227.11:4730 , piece_id=I6XK6IBGRE6RTB67BS6WYQG3YCYLSNSKS2CKIWGBU4YPTN2ZRY2Q, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.414450344 (564h2m40.28553338s) [noise=true , address=45.91.116.12:4228 , piece_id=LTJD3KG2O6GST5OVD4KI7HFAOOO5HJXRRETMBSEOUGN2CMOG5CTA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.415577741 (564h2m40.286660776s) [noise=true , address=145.79.227.32:4834 , piece_id=2YGMRVRYGLKI6ZMTFOZCX27XPXV3ZYB5RPOWZE46NIQFDRFQZ4CA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.416682445 (564h2m40.28776548s) [noise=true , address=45.91.116.11:4118 , piece_id=HUYHCJPUL3GAL3TF3H4TZ65OS6YWNDGND4K6KYKJWKMIKREKXVHA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.417003255 (564h2m40.28808629s) [noise=true , address=145.79.227.43:4158 , piece_id=VCDQ6327BCGUDSH56LO6D6KE7ZQUTOPX2J4PCTY3NCCGDVC2DZOQ, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.418148246 (564h2m40.289231321s) [noise=true , address=145.79.227.47:4267 , piece_id=XJP43VGLYA3UZHVCZNAQU36KH6VWVIJ5A7SG54JL7UTDPS6XBDYA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.418174656 (564h2m40.289257691s) [noise=true , address=45.91.116.5:4300 , piece_id=HL2XWR7XYFWW7K5H634EQ3EZHP2PEMVMVVMDKJ75TJ3XBT4RQGKA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.419339374 (564h2m40.290422409s) [noise=true , address=45.91.116.9:4264 , piece_id=N6OBFWQGNCFXHJ2RWVEZROZOOCPLBS4NTQAQNO3I5KW7I4RLFL6Q, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.420371149 (564h2m40.291454184s) [noise=true , address=145.79.227.37:4565 , piece_id=WJWLDBH2U2QVP6ELF4Y2PCIBDKTMGHU7W4MY3PTXB3YKIADEJKNA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:42.017668772 (564h2m41.888751807s) [upload=494534]: Next returned nil segment Dec 29 03:31:42.017681617 (564h2m41.888764662s) [upload=494534]: Waiting for error group managing segments... Dec 29 03:31:42.391922515 (564h2m42.26300556s) [noise=true , address=145.79.227.18:4006 , piece_id=BWFQ24R7OSSOA3XHBDI24VJDYJOARW6JVRI5OAFR7IKNX4TVFWFA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.408585584 (564h2m42.279668619s) [noise=true , address=145.79.227.32:4834 , piece_id=2YGMRVRYGLKI6ZMTFOZCX27XPXV3ZYB5RPOWZE46NIQFDRFQZ4CA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.412644553 (564h2m42.283727598s) [noise=true , address=145.79.227.37:4565 , piece_id=WJWLDBH2U2QVP6ELF4Y2PCIBDKTMGHU7W4MY3PTXB3YKIADEJKNA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.425607085 (564h2m42.29669012s) [noise=true , address=145.79.227.47:4267 , piece_id=XJP43VGLYA3UZHVCZNAQU36KH6VWVIJ5A7SG54JL7UTDPS6XBDYA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.431056222 (564h2m42.302139257s) [noise=true , address=145.79.227.11:4730 , piece_id=I6XK6IBGRE6RTB67BS6WYQG3YCYLSNSKS2CKIWGBU4YPTN2ZRY2Q, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.590366150 (564h2m42.461449185s) [noise=true , address=145.79.227.6:4259 , piece_id=ASTLNUGO5PX5DB5ALAQ3SCXIGMTRJIQCWNG5WFXA5OQN4HL2ZTXA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.643846905 (564h2m42.51492995s) [noise=true , address=45.91.116.9:4264 , piece_id=N6OBFWQGNCFXHJ2RWVEZROZOOCPLBS4NTQAQNO3I5KW7I4RLFL6Q, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.650832878 (564h2m42.521915933s) [noise=true , address=77.90.31.12:4589 , piece_id=LVXDUSPX73DWQHEK4Z5WH7NYQTDYFTAM2ALLVCHNDCLHECQ7PWKQ, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.661378509 (564h2m42.532461544s) [noise=true , address=45.91.116.14:4189 , piece_id=OZLPMBLJICM5NOPAEBABB3YXYUGUJUB5IKGY2OE7GNNWE5UQA4XQ, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.666198777 (564h2m42.537281812s) [noise=true , address=77.90.31.10:4520 , piece_id=AEG4JRHDZISDIA776YW32SFHVP2B5MHL3N2VAA2TVJK5N26XUIKA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.995773675 (564h2m42.86685672s) [noise=true , address=77.90.31.13:4629 , piece_id=OWG2ZNE3UPFRYNUEDBW2D3732OO6ZSNNZ3ILHJUZUPXAQSN2RFMA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:43.030874201 (564h2m42.901957236s) [noise=true , address=45.91.116.11:4118 , piece_id=HUYHCJPUL3GAL3TF3H4TZ65OS6YWNDGND4K6KYKJWKMIKREKXVHA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:43.403788173 (564h2m43.274871218s) [noise=true , address=45.91.116.6:4973 , piece_id=BXPX4L6XFGAHAMZ6G6Q27D7VJOOBGIBPWUOJXC3EFN6RXCLL2VVA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:43.403812419 (564h2m43.274895464s) [seg_pos={0 0}, upload=494534]: Segment reached optimal threshold of 13 pieces. Dec 29 03:31:43.404082474 (564h2m43.275165549s) [noise=true , address=145.79.227.43:4158 , piece_id=VCDQ6327BCGUDSH56LO6D6KE7ZQUTOPX2J4PCTY3NCCGDVC2DZOQ, seg_pos={0 0}, upload=494534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 29 03:31:43.404094617 (564h2m43.275177662s) [noise=true , address=145.79.227.43:4158 , piece_id=VCDQ6327BCGUDSH56LO6D6KE7ZQUTOPX2J4PCTY3NCCGDVC2DZOQ, seg_pos={0 0}, upload=494534]: Long tail cancellation due to optimal threshold Dec 29 03:31:43.404310327 (564h2m43.275393362s) [noise=true , address=45.91.116.12:4228 , piece_id=LTJD3KG2O6GST5OVD4KI7HFAOOO5HJXRRETMBSEOUGN2CMOG5CTA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:38814->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:38814->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38814->45.91.116.12:4228: use of closed network connection Dec 29 03:31:43.404339703 (564h2m43.275422748s) [noise=true , address=45.91.116.12:4228 , piece_id=LTJD3KG2O6GST5OVD4KI7HFAOOO5HJXRRETMBSEOUGN2CMOG5CTA, seg_pos={0 0}, upload=494534]: Long tail cancellation due to optimal threshold Dec 29 03:31:43.404392834 (564h2m43.275475869s) [noise=true , address=45.91.116.5:4300 , piece_id=HL2XWR7XYFWW7K5H634EQ3EZHP2PEMVMVVMDKJ75TJ3XBT4RQGKA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:49458->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:49458->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49458->45.91.116.5:4300: use of closed network connection Dec 29 03:31:43.404401231 (564h2m43.275484266s) [noise=true , address=45.91.116.5:4300 , piece_id=HL2XWR7XYFWW7K5H634EQ3EZHP2PEMVMVVMDKJ75TJ3XBT4RQGKA, seg_pos={0 0}, upload=494534]: Long tail cancellation due to optimal threshold Dec 29 03:31:43.404420036 (564h2m43.275503071s) [seg_pos={0 0}, upload=494534]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:31:43.404491923 (564h2m43.275574978s) [upload=494534]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:31:43.404497894 (564h2m43.27558095s) [upload=494534]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:31:43.695226022 (564h2m43.566309057s) [upload=494534]: Done uploading segments... Dec 29 03:31:43.695245209 (564h2m43.566328254s) [upload=494534]: Upload finished. err: <nil> Duration: 6.01920894s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [X....X.X.XX..XX.....XX.X.....X.X..XXXX..XXX.XX...X.X.....XXX] duration: 30.002248056s trace: https://jaeger.dev.storj.io/trace/70603e9358c10f6c FAIL (2025-12-29 03:32:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: OZLPMBLJICM5NOPAEBABB3YXYUGUJUB5IKGY2OE7GNNWE5UQA4XQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: OWG2ZNE3UPFRYNUEDBW2D3732OO6ZSNNZ3ILHJUZUPXAQSN2RFMA) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: LVXDUSPX73DWQHEK4Z5WH7NYQTDYFTAM2ALLVCHNDCLHECQ7PWKQ) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: BXPX4L6XFGAHAMZ6G6Q27D7VJOOBGIBPWUOJXC3EFN6RXCLL2VVA) context deadline exceeded ======LOGS====== 4 (564h2m40.033183739s) [upload=494534]: Uploading segments... Dec 29 03:31:40.162237635 (564h2m40.03332067s) [upload=494534]: Got next segment. Inline: false Dec 29 03:31:40.162275738 (564h2m40.033358773s) [upload=494534]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:31:40.162280206 (564h2m40.033363241s) [upload=494534]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:31:40.403193059 (564h2m40.274276094s) [seg_pos={0 0}, upload=494534]: Begin upload segment... Dec 29 03:31:40.403243144 (564h2m40.274326179s) [seg_pos={0 0}, upload=494534]: Done begin upload segment. Dec 29 03:31:40.403269044 (564h2m40.274352079s) [noise=true , address=77.90.31.10:4520 , piece_id=AEG4JRHDZISDIA776YW32SFHVP2B5MHL3N2VAA2TVJK5N26XUIKA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.407207904 (564h2m40.278290939s) [noise=true , address=45.91.116.6:4973 , piece_id=BXPX4L6XFGAHAMZ6G6Q27D7VJOOBGIBPWUOJXC3EFN6RXCLL2VVA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.408387570 (564h2m40.279470605s) [noise=true , address=145.79.227.18:4006 , piece_id=BWFQ24R7OSSOA3XHBDI24VJDYJOARW6JVRI5OAFR7IKNX4TVFWFA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.409424213 (564h2m40.280507248s) [noise=true , address=45.91.116.14:4189 , piece_id=OZLPMBLJICM5NOPAEBABB3YXYUGUJUB5IKGY2OE7GNNWE5UQA4XQ, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.410016982 (564h2m40.281100017s) [noise=true , address=77.90.31.13:4629 , piece_id=OWG2ZNE3UPFRYNUEDBW2D3732OO6ZSNNZ3ILHJUZUPXAQSN2RFMA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.411153306 (564h2m40.282236351s) [noise=true , address=77.90.31.12:4589 , piece_id=LVXDUSPX73DWQHEK4Z5WH7NYQTDYFTAM2ALLVCHNDCLHECQ7PWKQ, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.412201522 (564h2m40.283284557s) [noise=true , address=145.79.227.6:4259 , piece_id=ASTLNUGO5PX5DB5ALAQ3SCXIGMTRJIQCWNG5WFXA5OQN4HL2ZTXA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.413316305 (564h2m40.28439935s) [noise=true , address=145.79.227.11:4730 , piece_id=I6XK6IBGRE6RTB67BS6WYQG3YCYLSNSKS2CKIWGBU4YPTN2ZRY2Q, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.414450344 (564h2m40.28553338s) [noise=true , address=45.91.116.12:4228 , piece_id=LTJD3KG2O6GST5OVD4KI7HFAOOO5HJXRRETMBSEOUGN2CMOG5CTA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.415577741 (564h2m40.286660776s) [noise=true , address=145.79.227.32:4834 , piece_id=2YGMRVRYGLKI6ZMTFOZCX27XPXV3ZYB5RPOWZE46NIQFDRFQZ4CA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.416682445 (564h2m40.28776548s) [noise=true , address=45.91.116.11:4118 , piece_id=HUYHCJPUL3GAL3TF3H4TZ65OS6YWNDGND4K6KYKJWKMIKREKXVHA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.417003255 (564h2m40.28808629s) [noise=true , address=145.79.227.43:4158 , piece_id=VCDQ6327BCGUDSH56LO6D6KE7ZQUTOPX2J4PCTY3NCCGDVC2DZOQ, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.418148246 (564h2m40.289231321s) [noise=true , address=145.79.227.47:4267 , piece_id=XJP43VGLYA3UZHVCZNAQU36KH6VWVIJ5A7SG54JL7UTDPS6XBDYA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.418174656 (564h2m40.289257691s) [noise=true , address=45.91.116.5:4300 , piece_id=HL2XWR7XYFWW7K5H634EQ3EZHP2PEMVMVVMDKJ75TJ3XBT4RQGKA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.419339374 (564h2m40.290422409s) [noise=true , address=45.91.116.9:4264 , piece_id=N6OBFWQGNCFXHJ2RWVEZROZOOCPLBS4NTQAQNO3I5KW7I4RLFL6Q, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:40.420371149 (564h2m40.291454184s) [noise=true , address=145.79.227.37:4565 , piece_id=WJWLDBH2U2QVP6ELF4Y2PCIBDKTMGHU7W4MY3PTXB3YKIADEJKNA, seg_pos={0 0}, upload=494534]: Uploading piece... Dec 29 03:31:42.017668772 (564h2m41.888751807s) [upload=494534]: Next returned nil segment Dec 29 03:31:42.017681617 (564h2m41.888764662s) [upload=494534]: Waiting for error group managing segments... Dec 29 03:31:42.391922515 (564h2m42.26300556s) [noise=true , address=145.79.227.18:4006 , piece_id=BWFQ24R7OSSOA3XHBDI24VJDYJOARW6JVRI5OAFR7IKNX4TVFWFA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.408585584 (564h2m42.279668619s) [noise=true , address=145.79.227.32:4834 , piece_id=2YGMRVRYGLKI6ZMTFOZCX27XPXV3ZYB5RPOWZE46NIQFDRFQZ4CA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.412644553 (564h2m42.283727598s) [noise=true , address=145.79.227.37:4565 , piece_id=WJWLDBH2U2QVP6ELF4Y2PCIBDKTMGHU7W4MY3PTXB3YKIADEJKNA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.425607085 (564h2m42.29669012s) [noise=true , address=145.79.227.47:4267 , piece_id=XJP43VGLYA3UZHVCZNAQU36KH6VWVIJ5A7SG54JL7UTDPS6XBDYA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.431056222 (564h2m42.302139257s) [noise=true , address=145.79.227.11:4730 , piece_id=I6XK6IBGRE6RTB67BS6WYQG3YCYLSNSKS2CKIWGBU4YPTN2ZRY2Q, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.590366150 (564h2m42.461449185s) [noise=true , address=145.79.227.6:4259 , piece_id=ASTLNUGO5PX5DB5ALAQ3SCXIGMTRJIQCWNG5WFXA5OQN4HL2ZTXA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.643846905 (564h2m42.51492995s) [noise=true , address=45.91.116.9:4264 , piece_id=N6OBFWQGNCFXHJ2RWVEZROZOOCPLBS4NTQAQNO3I5KW7I4RLFL6Q, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.650832878 (564h2m42.521915933s) [noise=true , address=77.90.31.12:4589 , piece_id=LVXDUSPX73DWQHEK4Z5WH7NYQTDYFTAM2ALLVCHNDCLHECQ7PWKQ, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.661378509 (564h2m42.532461544s) [noise=true , address=45.91.116.14:4189 , piece_id=OZLPMBLJICM5NOPAEBABB3YXYUGUJUB5IKGY2OE7GNNWE5UQA4XQ, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.666198777 (564h2m42.537281812s) [noise=true , address=77.90.31.10:4520 , piece_id=AEG4JRHDZISDIA776YW32SFHVP2B5MHL3N2VAA2TVJK5N26XUIKA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:42.995773675 (564h2m42.86685672s) [noise=true , address=77.90.31.13:4629 , piece_id=OWG2ZNE3UPFRYNUEDBW2D3732OO6ZSNNZ3ILHJUZUPXAQSN2RFMA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:43.030874201 (564h2m42.901957236s) [noise=true , address=45.91.116.11:4118 , piece_id=HUYHCJPUL3GAL3TF3H4TZ65OS6YWNDGND4K6KYKJWKMIKREKXVHA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:43.403788173 (564h2m43.274871218s) [noise=true , address=45.91.116.6:4973 , piece_id=BXPX4L6XFGAHAMZ6G6Q27D7VJOOBGIBPWUOJXC3EFN6RXCLL2VVA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: <nil> Dec 29 03:31:43.403812419 (564h2m43.274895464s) [seg_pos={0 0}, upload=494534]: Segment reached optimal threshold of 13 pieces. Dec 29 03:31:43.404082474 (564h2m43.275165549s) [noise=true , address=145.79.227.43:4158 , piece_id=VCDQ6327BCGUDSH56LO6D6KE7ZQUTOPX2J4PCTY3NCCGDVC2DZOQ, seg_pos={0 0}, upload=494534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 29 03:31:43.404094617 (564h2m43.275177662s) [noise=true , address=145.79.227.43:4158 , piece_id=VCDQ6327BCGUDSH56LO6D6KE7ZQUTOPX2J4PCTY3NCCGDVC2DZOQ, seg_pos={0 0}, upload=494534]: Long tail cancellation due to optimal threshold Dec 29 03:31:43.404310327 (564h2m43.275393362s) [noise=true , address=45.91.116.12:4228 , piece_id=LTJD3KG2O6GST5OVD4KI7HFAOOO5HJXRRETMBSEOUGN2CMOG5CTA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:38814->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:38814->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38814->45.91.116.12:4228: use of closed network connection Dec 29 03:31:43.404339703 (564h2m43.275422748s) [noise=true , address=45.91.116.12:4228 , piece_id=LTJD3KG2O6GST5OVD4KI7HFAOOO5HJXRRETMBSEOUGN2CMOG5CTA, seg_pos={0 0}, upload=494534]: Long tail cancellation due to optimal threshold Dec 29 03:31:43.404392834 (564h2m43.275475869s) [noise=true , address=45.91.116.5:4300 , piece_id=HL2XWR7XYFWW7K5H634EQ3EZHP2PEMVMVVMDKJ75TJ3XBT4RQGKA, seg_pos={0 0}, upload=494534]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:49458->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:49458->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49458->45.91.116.5:4300: use of closed network connection Dec 29 03:31:43.404401231 (564h2m43.275484266s) [noise=true , address=45.91.116.5:4300 , piece_id=HL2XWR7XYFWW7K5H634EQ3EZHP2PEMVMVVMDKJ75TJ3XBT4RQGKA, seg_pos={0 0}, upload=494534]: Long tail cancellation due to optimal threshold Dec 29 03:31:43.404420036 (564h2m43.275503071s) [seg_pos={0 0}, upload=494534]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:31:43.404491923 (564h2m43.275574978s) [upload=494534]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:31:43.404497894 (564h2m43.27558095s) [upload=494534]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:31:43.695226022 (564h2m43.566309057s) [upload=494534]: Done uploading segments... Dec 29 03:31:43.695245209 (564h2m43.566328254s) [upload=494534]: Upload finished. err: <nil> Duration: 30.002270699s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............................................XX..............] duration: 3.538931174s trace: https://jaeger.dev.storj.io/trace/481fdf974580574a FAIL (2025-12-29 03:31:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.538957365s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 24.345994931s trace: https://jaeger.dev.storj.io/trace/1584175ba8dba7fe FAIL (2025-12-29 03:31:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ece... Dec 29 03:20:05.592697598 (563h51m5.463780643s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593249339 (563h51m5.464332384s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593687443 (563h51m5.464770488s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594173598 (563h51m5.465256633s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594639957 (563h51m5.465723002s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595331994 (563h51m5.466415039s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595834151 (563h51m5.466917206s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.596329844 (563h51m5.467412899s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599128643 (563h51m5.470211688s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599196953 (563h51m5.470279988s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.601094367 (563h51m5.472177412s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.602868896 (563h51m5.473951941s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.604661328 (563h51m5.475744374s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.606462138 (563h51m5.477545183s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.609580124 (563h51m5.480663169s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.611644807 (563h51m5.482727852s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:06.901416710 (563h51m6.772499755s) [upload=494362]: Next returned nil segment Dec 29 03:20:06.901434123 (563h51m6.772517168s) [upload=494362]: Waiting for error group managing segments... Dec 29 03:20:07.237383793 (563h51m7.108466848s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.439111607 (563h51m7.310194652s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.462301153 (563h51m7.333384198s) [noise=true , address=145.79.227.21:4498 , piece_id=RTK3PTWGWTIQSNCGREQB37OID2RGK37QWEKTDYFQKASGH6SZCURA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.479891718 (563h51m7.350974763s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.486062297 (563h51m7.357145342s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.578667930 (563h51m7.449750975s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.701241555 (563h51m7.5723246s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.745868671 (563h51m7.616951706s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.799936903 (563h51m7.671019948s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.430929511 (563h51m8.302012556s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.432106221 (563h51m8.303189256s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.447633778 (563h51m8.318716813s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.459472520 (563h51m8.330555555s) [noise=true , address=145.79.227.34:4916 , piece_id=4U67LINR5A4GR2NIUVMB2IPTT5DJQOYHM6HDYKZEICKF5JY53VFA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499215165 (563h51m8.37029821s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499245653 (563h51m8.370328698s) [seg_pos={0 0}, upload=494362]: Segment reached optimal threshold of 14 pieces. Dec 29 03:20:08.499544722 (563h51m8.370627797s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.499555243 (563h51m8.370638288s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.499833272 (563h51m8.370916327s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection Dec 29 03:20:08.499843291 (563h51m8.370926366s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500107964 (563h51m8.371191019s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.500166165 (563h51m8.37124921s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500868092 (563h51m8.371951147s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53): protocol: context canceled; context canceled Dec 29 03:20:08.500880466 (563h51m8.371963511s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500945178 (563h51m8.372028223s) [seg_pos={0 0}, upload=494362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:20:08.501128358 (563h51m8.372211413s) [upload=494362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:20:08.501156331 (563h51m8.372239376s) [upload=494362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:20:08.795508553 (563h51m8.666591598s) [upload=494362]: Done uploading segments... Dec 29 03:20:08.795532639 (563h51m8.666615684s) [upload=494362]: Upload finished. err: <nil> Duration: 24.34602066s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.X....X.X.XX..XX.....XX.X.....X.X..XXXX..XXX.XX...X.X.....XX] duration: 30.002911898s trace: https://jaeger.dev.storj.io/trace/3827d7cc75f5ac3d FAIL (2025-12-29 03:31:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: MITOI4ECXD332RT275VEFRRMDIJDRHVFVHY2BJEV4LTBTXP2OS4Q) context deadline exceeded; piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: XMWG3WGJPDLCTYRFBMTUDAD4RTBXJVF4L2BI7W3TRHUXNH3MQDHQ) context deadline exceeded ======LOGS====== Dec 29 03:30:43.238069103 (564h1m43.109152148s) [upload=494519]: Starting upload Dec 29 03:30:43.238140439 (564h1m43.109223474s) [upload=494519]: Done starting upload Dec 29 03:30:43.238346962 (564h1m43.109429997s) [upload=494519]: Uploading segments... Dec 29 03:30:43.238433658 (564h1m43.109516703s) [upload=494519]: Got next segment. Inline: false Dec 29 03:30:43.238466059 (564h1m43.109549095s) [upload=494519]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:30:43.238471470 (564h1m43.109554505s) [upload=494519]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:30:43.498386049 (564h1m43.369469084s) [seg_pos={0 0}, upload=494519]: Begin upload segment... Dec 29 03:30:43.498436755 (564h1m43.369519791s) [seg_pos={0 0}, upload=494519]: Done begin upload segment. Dec 29 03:30:43.498465971 (564h1m43.369549016s) [noise=true , address=45.91.116.12:4228 , piece_id=XMWG3WGJPDLCTYRFBMTUDAD4RTBXJVF4L2BI7W3TRHUXNH3MQDHQ, seg_pos={0 0}, upload=494519]: Uploading piece... Dec 29 03:30:43.499298437 (564h1m43.370381492s) [noise=true , address=145.79.227.40:4051 , piece_id=JVM7IPTLOBCSD5M4EFKXMXV4WBRQLRGLL7DBZ3FYHIFHKWCXRYTA, seg_pos={0 0}, upload=494519]: Uploading piece... Dec 29 03:30:43.499757881 (564h1m43.370840916s) [noise=true , address=145.79.227.7:4293 , piece_id=QF76422GKZBM2TEAEAYLFGQEIJITAHXSWKMQQTJOT4M5MCEACZFA, seg_pos={0 0}, upload=494519]: Uploading piece... Dec 29 03:30:43.500132034 (564h1m43.371215069s) [noise=true , address=145.79.227.50:4375 , piece_id=4XRC5S3ZLO3V3Y45QOGDLDHWQIFW2AOGIUCKJN4Y7YURFSUK4R4A, seg_pos={0 0}, upload=494519]: Uploading piece... Dec 29 03:30:43.500414532 (564h1m43.371497567s) [noise=true , address=145.79.227.51:4410 , piece_id=N5SEDJHTZZ5VY2TCWKKZXC2TINFLYDIBEEWEZTKZI634XGSI5JOA, seg_pos={0 0}, upload=494519]: Uploading piece... Dec 29 03:30:43.500693393 (564h1m43.371776428s) [noise=true , address=145.79.227.48:4301 , piece_id=GI6HQC3ZYOH7LFFUAAPVKAPTX5R2KIOFWAFP4T46I3GOVVY3TEOQ, seg_pos={0 0}, upload=494519]: Uploading piece... Dec 29 03:30:43.500992082 (564h1m43.372075117s) [noise=true , address=145.79.227.47:4265 , piece_id=272MWUCZEUVQFUE3DFQHLAC6GKOU4KHCLWOYCIB6HENN5YFSK2JA, seg_pos={0 0}, upload=494519]: Uploading piece... Dec 29 03:30:43.501284328 (564h1m43.372367363s) [noise=true , address=77.90.31.19:4841 , piece_id=MITOI4ECXD332RT275VEFRRMDIJDRHVFVHY2BJEV4LTBTXP2OS4Q, seg_pos={0 0}, upload=494519]: Uploading piece... Dec 29 03:30:43.501594429 (564h1m43.372677464s) [noise=true , address=77.90.31.14:4661 , piece_id=HRORJDE3WW6EPTDH4OUQBJMJ4ZJJ2N2W4W56BIMHHZLF633W47PQ, seg_pos={0 0}, upload=494519]: Uploading piece... Dec 29 03:30:43.502006103 (564h1m43.373089138s) [noise=true , address=145.79.227.5:4223 , piece_id=TRHTPQH7YGRUHPHECPVKFXR2WRHG7HFEBNKBJVBGSQPZYE3WK2JQ, seg_pos={0 0}, upload=494519]: Uploading piece... Dec 29 03:30:43.502868484 (564h1m43.373951519s) [noise=true , address=145.79.227.13:4803 , piece_id=KW7F7VVT6SUQPJKPHNYYAKKJBTRR4EUABTFFDOWAPW54OIV6HWMQ, seg_pos={0 0}, upload=494519]: Uploading piece... Dec 29 03:30:43.503269528 (564h1m43.374352573s) [noise=true , address=145.79.227.16:4911 , piece_id=DHHUOKE3J6UIMPXJIDF7XFTJ6AFML4CWGCL7UWFYSTEDLWJK4GWQ, seg_pos={0 0}, upload=494519]: Uploading piece... Dec 29 03:30:43.503907402 (564h1m43.374990447s) [noise=true , address=145.79.227.15:4875 , piece_id=CGM4KY3VN4EACJTTL53J234BDEDT7B2JFZPJJQFNKO2XAEMVWEOQ, seg_pos={0 0}, upload=494519]: Uploading piece... Dec 29 03:30:43.505050990 (564h1m43.376134035s) [noise=true , address=77.90.31.13:4626 , piece_id=VCSX7C76U7IA4X3U3IITPCJUXT5YNQST3HMVVZJ4Q4DNYG5QTAXQ, seg_pos={0 0}, upload=494519]: Uploading piece... Dec 29 03:30:43.507264085 (564h1m43.37834713s) [noise=true , address=145.79.227.14:4838 , piece_id=2Z6BAR3AM2UPJAUKQ7STBHSCQMXMVJ2XOMHX5W2IA7XZ7GXI4YPA, seg_pos={0 0}, upload=494519]: Uploading piece... Dec 29 03:30:43.508957769 (564h1m43.380040814s) [noise=true , address=145.79.227.11:4729 , piece_id=QYOGBVZV27KVOBIHH3HSFMTZNIU2QDCUKW64QXM2FQRFGU4PRMSA, seg_pos={0 0}, upload=494519]: Uploading piece... Dec 29 03:30:45.197419605 (564h1m45.06850265s) [upload=494519]: Next returned nil segment Dec 29 03:30:45.197435445 (564h1m45.06851848s) [upload=494519]: Waiting for error group managing segments... Dec 29 03:30:45.477986829 (564h1m45.349069864s) [noise=true , address=145.79.227.51:4410 , piece_id=N5SEDJHTZZ5VY2TCWKKZXC2TINFLYDIBEEWEZTKZI634XGSI5JOA, seg_pos={0 0}, upload=494519]: Done uploading piece. err: <nil> Dec 29 03:30:45.479112532 (564h1m45.350195577s) [noise=true , address=145.79.227.48:4301 , piece_id=GI6HQC3ZYOH7LFFUAAPVKAPTX5R2KIOFWAFP4T46I3GOVVY3TEOQ, seg_pos={0 0}, upload=494519]: Done uploading piece. err: <nil> Dec 29 03:30:45.485858638 (564h1m45.356941673s) [noise=true , address=145.79.227.13:4803 , piece_id=KW7F7VVT6SUQPJKPHNYYAKKJBTRR4EUABTFFDOWAPW54OIV6HWMQ, seg_pos={0 0}, upload=494519]: Done uploading piece. err: <nil> Dec 29 03:30:45.507419402 (564h1m45.378502447s) [noise=true , address=145.79.227.50:4375 , piece_id=4XRC5S3ZLO3V3Y45QOGDLDHWQIFW2AOGIUCKJN4Y7YURFSUK4R4A, seg_pos={0 0}, upload=494519]: Done uploading piece. err: <nil> Dec 29 03:30:45.507508672 (564h1m45.378591737s) [noise=true , address=145.79.227.15:4875 , piece_id=CGM4KY3VN4EACJTTL53J234BDEDT7B2JFZPJJQFNKO2XAEMVWEOQ, seg_pos={0 0}, upload=494519]: Done uploading piece. err: <nil> Dec 29 03:30:45.509276008 (564h1m45.380359053s) [noise=true , address=145.79.227.11:4729 , piece_id=QYOGBVZV27KVOBIHH3HSFMTZNIU2QDCUKW64QXM2FQRFGU4PRMSA, seg_pos={0 0}, upload=494519]: Done uploading piece. err: <nil> Dec 29 03:30:45.509466129 (564h1m45.380549164s) [noise=true , address=145.79.227.7:4293 , piece_id=QF76422GKZBM2TEAEAYLFGQEIJITAHXSWKMQQTJOT4M5MCEACZFA, seg_pos={0 0}, upload=494519]: Done uploading piece. err: <nil> Dec 29 03:30:45.509699674 (564h1m45.380782709s) [noise=true , address=145.79.227.16:4911 , piece_id=DHHUOKE3J6UIMPXJIDF7XFTJ6AFML4CWGCL7UWFYSTEDLWJK4GWQ, seg_pos={0 0}, upload=494519]: Done uploading piece. err: <nil> Dec 29 03:30:45.509965490 (564h1m45.381048535s) [noise=true , address=145.79.227.5:4223 , piece_id=TRHTPQH7YGRUHPHECPVKFXR2WRHG7HFEBNKBJVBGSQPZYE3WK2JQ, seg_pos={0 0}, upload=494519]: Done uploading piece. err: <nil> Dec 29 03:30:45.514431544 (564h1m45.385514589s) [noise=true , address=145.79.227.14:4838 , piece_id=2Z6BAR3AM2UPJAUKQ7STBHSCQMXMVJ2XOMHX5W2IA7XZ7GXI4YPA, seg_pos={0 0}, upload=494519]: Done uploading piece. err: <nil> Dec 29 03:30:45.528740210 (564h1m45.399823245s) [noise=true , address=145.79.227.47:4265 , piece_id=272MWUCZEUVQFUE3DFQHLAC6GKOU4KHCLWOYCIB6HENN5YFSK2JA, seg_pos={0 0}, upload=494519]: Done uploading piece. err: <nil> Dec 29 03:30:45.772021909 (564h1m45.643104944s) [noise=true , address=45.91.116.12:4228 , piece_id=XMWG3WGJPDLCTYRFBMTUDAD4RTBXJVF4L2BI7W3TRHUXNH3MQDHQ, seg_pos={0 0}, upload=494519]: Done uploading piece. err: <nil> Dec 29 03:30:45.809238664 (564h1m45.680321699s) [noise=true , address=77.90.31.19:4841 , piece_id=MITOI4ECXD332RT275VEFRRMDIJDRHVFVHY2BJEV4LTBTXP2OS4Q, seg_pos={0 0}, upload=494519]: Done uploading piece. err: <nil> Dec 29 03:30:45.809273901 (564h1m45.680356936s) [seg_pos={0 0}, upload=494519]: Segment reached optimal threshold of 13 pieces. Dec 29 03:30:45.809578611 (564h1m45.680661656s) [noise=true , address=77.90.31.13:4626 , piece_id=VCSX7C76U7IA4X3U3IITPCJUXT5YNQST3HMVVZJ4Q4DNYG5QTAXQ, seg_pos={0 0}, upload=494519]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Dec 29 03:30:45.809587999 (564h1m45.680671044s) [noise=true , address=77.90.31.13:4626 , piece_id=VCSX7C76U7IA4X3U3IITPCJUXT5YNQST3HMVVZJ4Q4DNYG5QTAXQ, seg_pos={0 0}, upload=494519]: Long tail cancellation due to optimal threshold Dec 29 03:30:45.809704540 (564h1m45.680787585s) [noise=true , address=77.90.31.14:4661 , piece_id=HRORJDE3WW6EPTDH4OUQBJMJ4ZJJ2N2W4W56BIMHHZLF633W47PQ, seg_pos={0 0}, upload=494519]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Dec 29 03:30:45.809740489 (564h1m45.680823524s) [noise=true , address=77.90.31.14:4661 , piece_id=HRORJDE3WW6EPTDH4OUQBJMJ4ZJJ2N2W4W56BIMHHZLF633W47PQ, seg_pos={0 0}, upload=494519]: Long tail cancellation due to optimal threshold Dec 29 03:30:45.809948004 (564h1m45.681031039s) [noise=true , address=145.79.227.40:4051 , piece_id=JVM7IPTLOBCSD5M4EFKXMXV4WBRQLRGLL7DBZ3FYHIFHKWCXRYTA, seg_pos={0 0}, upload=494519]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 29 03:30:45.809991167 (564h1m45.681074202s) [noise=true , address=145.79.227.40:4051 , piece_id=JVM7IPTLOBCSD5M4EFKXMXV4WBRQLRGLL7DBZ3FYHIFHKWCXRYTA, seg_pos={0 0}, upload=494519]: Long tail cancellation due to optimal threshold Dec 29 03:30:45.810022285 (564h1m45.68110532s) [seg_pos={0 0}, upload=494519]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:30:45.810107047 (564h1m45.681190092s) [upload=494519]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:30:45.810114922 (564h1m45.681197957s) [upload=494519]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:30:46.105682279 (564h1m45.976765324s) [upload=494519]: Done uploading segments... Dec 29 03:30:46.105756730 (564h1m45.976839765s) [upload=494519]: Upload finished. err: <nil> Duration: 30.002933309s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 30.000155129s trace: https://jaeger.dev.storj.io/trace/1584175ba8dba7fe FAIL (2025-12-29 03:31:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ece... Dec 29 03:20:05.592697598 (563h51m5.463780643s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593249339 (563h51m5.464332384s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593687443 (563h51m5.464770488s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594173598 (563h51m5.465256633s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594639957 (563h51m5.465723002s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595331994 (563h51m5.466415039s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595834151 (563h51m5.466917206s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.596329844 (563h51m5.467412899s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599128643 (563h51m5.470211688s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599196953 (563h51m5.470279988s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.601094367 (563h51m5.472177412s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.602868896 (563h51m5.473951941s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.604661328 (563h51m5.475744374s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.606462138 (563h51m5.477545183s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.609580124 (563h51m5.480663169s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.611644807 (563h51m5.482727852s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:06.901416710 (563h51m6.772499755s) [upload=494362]: Next returned nil segment Dec 29 03:20:06.901434123 (563h51m6.772517168s) [upload=494362]: Waiting for error group managing segments... Dec 29 03:20:07.237383793 (563h51m7.108466848s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.439111607 (563h51m7.310194652s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.462301153 (563h51m7.333384198s) [noise=true , address=145.79.227.21:4498 , piece_id=RTK3PTWGWTIQSNCGREQB37OID2RGK37QWEKTDYFQKASGH6SZCURA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.479891718 (563h51m7.350974763s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.486062297 (563h51m7.357145342s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.578667930 (563h51m7.449750975s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.701241555 (563h51m7.5723246s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.745868671 (563h51m7.616951706s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.799936903 (563h51m7.671019948s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.430929511 (563h51m8.302012556s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.432106221 (563h51m8.303189256s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.447633778 (563h51m8.318716813s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.459472520 (563h51m8.330555555s) [noise=true , address=145.79.227.34:4916 , piece_id=4U67LINR5A4GR2NIUVMB2IPTT5DJQOYHM6HDYKZEICKF5JY53VFA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499215165 (563h51m8.37029821s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499245653 (563h51m8.370328698s) [seg_pos={0 0}, upload=494362]: Segment reached optimal threshold of 14 pieces. Dec 29 03:20:08.499544722 (563h51m8.370627797s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.499555243 (563h51m8.370638288s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.499833272 (563h51m8.370916327s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection Dec 29 03:20:08.499843291 (563h51m8.370926366s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500107964 (563h51m8.371191019s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.500166165 (563h51m8.37124921s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500868092 (563h51m8.371951147s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53): protocol: context canceled; context canceled Dec 29 03:20:08.500880466 (563h51m8.371963511s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500945178 (563h51m8.372028223s) [seg_pos={0 0}, upload=494362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:20:08.501128358 (563h51m8.372211413s) [upload=494362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:20:08.501156331 (563h51m8.372239376s) [upload=494362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:20:08.795508553 (563h51m8.666591598s) [upload=494362]: Done uploading segments... Dec 29 03:20:08.795532639 (563h51m8.666615684s) [upload=494362]: Upload finished. err: <nil> Duration: 30.00018668s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............................................XX.............] duration: 3.910510143s trace: https://jaeger.dev.storj.io/trace/481fdf974580574a FAIL (2025-12-29 03:31:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.910527125s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [X.X....X.X.XX..XX.....XX.X.....X.X..XXXX..XXX.XX...X.X.....X] duration: 30.003523808s trace: https://jaeger.dev.storj.io/trace/15b4410c64f18fce FAIL (2025-12-29 03:30:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 03:30:02.538424530 (564h1m2.409507575s) [upload=494511]: Starting upload Dec 29 03:30:02.538542535 (564h1m2.40962558s) [upload=494511]: Done starting upload Dec 29 03:30:02.538764688 (564h1m2.409847733s) [upload=494511]: Uploading segments... Dec 29 03:30:02.538888443 (564h1m2.409971488s) [upload=494511]: Got next segment. Inline: false Dec 29 03:30:02.538941845 (564h1m2.41002489s) [upload=494511]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:30:02.538950471 (564h1m2.410033516s) [upload=494511]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:30:02.816464684 (564h1m2.687547739s) [seg_pos={0 0}, upload=494511]: Begin upload segment... Dec 29 03:30:02.816528024 (564h1m2.687611069s) [seg_pos={0 0}, upload=494511]: Done begin upload segment. Dec 29 03:30:02.816570826 (564h1m2.687653871s) [noise=true , address=145.79.227.11:4730 , piece_id=PTSDWJSESHAGT7YPGOHQNHLSIKEJIOB5TFSJLNEFLDIG7WQ5BRMA, seg_pos={0 0}, upload=494511]: Uploading piece... Dec 29 03:30:02.816850519 (564h1m2.687933574s) [noise=true , address=145.79.227.19:4077 , piece_id=CPMXMHDMFDT4TF3JN4XK3AQ4P3IEEIKAMWKDKYQFBER4SKNKHTDA, seg_pos={0 0}, upload=494511]: Uploading piece... Dec 29 03:30:02.817418781 (564h1m2.688501836s) [noise=true , address=145.79.227.50:4373 , piece_id=F446MJAV44FV5UWERNOAL3VGLBCQ7V2G3NJ3Y524YHX2U3EMPP5A, seg_pos={0 0}, upload=494511]: Uploading piece... Dec 29 03:30:02.817989828 (564h1m2.689072883s) [noise=true , address=77.90.31.18:4808 , piece_id=E3RVZMHBMHUJ4MSR44RAVMDVTD75KBRKVUFKTMSZKYXOO4CNFT6Q, seg_pos={0 0}, upload=494511]: Uploading piece... Dec 29 03:30:02.818458651 (564h1m2.689541706s) [noise=true , address=77.90.31.16:4735 , piece_id=426RMS3ER4DPENKIFUXCPQQDKEFG3GDEPRZAXTBKEVBPXQLN2WWA, seg_pos={0 0}, upload=494511]: Uploading piece... Dec 29 03:30:02.819168993 (564h1m2.690252048s) [noise=true , address=145.79.227.43:4159 , piece_id=UGEOANOJN333PQUVQY3LXEFN7ZUIRNGCJPZKUMUALIU6QSHWHHNA, seg_pos={0 0}, upload=494511]: Uploading piece... Dec 29 03:30:02.819702549 (564h1m2.690785594s) [noise=true , address=145.79.227.47:4267 , piece_id=X54JI7LFNLO4DPA3JRKD7XZDJSUGFQFBIUVQKMVFKURXCFK272YA, seg_pos={0 0}, upload=494511]: Uploading piece... Dec 29 03:30:02.819779556 (564h1m2.690862601s) [noise=true , address=45.91.116.7:4010 , piece_id=CCM3YISUJC2OTCA7PFINNHNDBE64GTNEOE7J5A2VLRE5ZPURGCTQ, seg_pos={0 0}, upload=494511]: Uploading piece... Dec 29 03:30:02.820225995 (564h1m2.69130904s) [noise=true , address=45.91.116.11:4120 , piece_id=IN5ATYACAMGX4MN3HLFR2D4H3P2NJT34EABUHEI4VGL4LWL7DJGA, seg_pos={0 0}, upload=494511]: Uploading piece... Dec 29 03:30:02.820242658 (564h1m2.691325703s) [noise=true , address=77.90.31.13:4626 , piece_id=4WG56YDS33OMACKJBQY566DZHA4LRV7NDLDUSP72TZANSF5OXZMQ, seg_pos={0 0}, upload=494511]: Uploading piece... Dec 29 03:30:02.820726800 (564h1m2.691809845s) [noise=true , address=45.91.116.12:4227 , piece_id=URBYIJUXJGOAZGEZQ6C4IWG57SIOWOG2D7MXG5ZIJSRITG7HNLXA, seg_pos={0 0}, upload=494511]: Uploading piece... Dec 29 03:30:02.820745374 (564h1m2.69182842s) [noise=true , address=45.91.116.5:4297 , piece_id=S5IH7YLIGDFSWWPOFS66MLLLM6T5KRC37ATBPTM2PJBDPUKFW5VA, seg_pos={0 0}, upload=494511]: Uploading piece... Dec 29 03:30:02.821175664 (564h1m2.692258709s) [noise=true , address=45.91.116.8:4047 , piece_id=IULMJMQL2DMS7JRA53MMWU2AX6KWL4V5UJASQIT7UN64NYGFQ7VQ, seg_pos={0 0}, upload=494511]: Uploading piece... Dec 29 03:30:02.821200551 (564h1m2.692283596s) [noise=true , address=77.90.31.12:4590 , piece_id=QSYQ4XVF2EUE52XL2QLGWEKGLRAN6EO2OAF5NGGLCJXEOA563OEQ, seg_pos={0 0}, upload=494511]: Uploading piece... Dec 29 03:30:02.821667851 (564h1m2.692750967s) [noise=true , address=45.91.116.9:4263 , piece_id=JCTZTTV4WTYQEXKOOUY6DHZNLUZT4MRU5JUPXTHVSSBOOCOHO4LA, seg_pos={0 0}, upload=494511]: Uploading piece... Dec 29 03:30:02.821676788 (564h1m2.692759833s) [noise=true , address=145.79.227.46:4601 , piece_id=XTSXORQ63SD2RJGBDHHX4XDAPFL4XELGWU5WZOOAV3M5EY3UD2BQ, seg_pos={0 0}, upload=494511]: Uploading piece... Dec 29 03:30:04.457675598 (564h1m4.328758663s) [upload=494511]: Next returned nil segment Dec 29 03:30:04.457693432 (564h1m4.328776477s) [upload=494511]: Waiting for error group managing segments... Dec 29 03:30:04.836499596 (564h1m4.707582641s) [noise=true , address=145.79.227.11:4730 , piece_id=PTSDWJSESHAGT7YPGOHQNHLSIKEJIOB5TFSJLNEFLDIG7WQ5BRMA, seg_pos={0 0}, upload=494511]: Done uploading piece. err: <nil> Dec 29 03:30:04.859861289 (564h1m4.730944334s) [noise=true , address=145.79.227.46:4601 , piece_id=XTSXORQ63SD2RJGBDHHX4XDAPFL4XELGWU5WZOOAV3M5EY3UD2BQ, seg_pos={0 0}, upload=494511]: Done uploading piece. err: <nil> Dec 29 03:30:04.898142420 (564h1m4.769225465s) [noise=true , address=145.79.227.47:4267 , piece_id=X54JI7LFNLO4DPA3JRKD7XZDJSUGFQFBIUVQKMVFKURXCFK272YA, seg_pos={0 0}, upload=494511]: Done uploading piece. err: <nil> Dec 29 03:30:04.909411759 (564h1m4.780494804s) [noise=true , address=145.79.227.19:4077 , piece_id=CPMXMHDMFDT4TF3JN4XK3AQ4P3IEEIKAMWKDKYQFBER4SKNKHTDA, seg_pos={0 0}, upload=494511]: Done uploading piece. err: <nil> Dec 29 03:30:05.048718477 (564h1m4.919801522s) [noise=true , address=77.90.31.13:4626 , piece_id=4WG56YDS33OMACKJBQY566DZHA4LRV7NDLDUSP72TZANSF5OXZMQ, seg_pos={0 0}, upload=494511]: Done uploading piece. err: <nil> Dec 29 03:30:05.084691974 (564h1m4.955775019s) [noise=true , address=45.91.116.8:4047 , piece_id=IULMJMQL2DMS7JRA53MMWU2AX6KWL4V5UJASQIT7UN64NYGFQ7VQ, seg_pos={0 0}, upload=494511]: Done uploading piece. err: <nil> Dec 29 03:30:05.120682234 (564h1m4.991765279s) [noise=true , address=77.90.31.16:4735 , piece_id=426RMS3ER4DPENKIFUXCPQQDKEFG3GDEPRZAXTBKEVBPXQLN2WWA, seg_pos={0 0}, upload=494511]: Done uploading piece. err: <nil> Dec 29 03:30:05.140000749 (564h1m5.011083804s) [noise=true , address=77.90.31.18:4808 , piece_id=E3RVZMHBMHUJ4MSR44RAVMDVTD75KBRKVUFKTMSZKYXOO4CNFT6Q, seg_pos={0 0}, upload=494511]: Done uploading piece. err: <nil> Dec 29 03:30:05.187259025 (564h1m5.05834207s) [noise=true , address=45.91.116.5:4297 , piece_id=S5IH7YLIGDFSWWPOFS66MLLLM6T5KRC37ATBPTM2PJBDPUKFW5VA, seg_pos={0 0}, upload=494511]: Done uploading piece. err: <nil> Dec 29 03:30:05.207672454 (564h1m5.078755509s) [noise=true , address=45.91.116.11:4120 , piece_id=IN5ATYACAMGX4MN3HLFR2D4H3P2NJT34EABUHEI4VGL4LWL7DJGA, seg_pos={0 0}, upload=494511]: Done uploading piece. err: <nil> Dec 29 03:30:05.229462195 (564h1m5.10054527s) [noise=true , address=45.91.116.9:4263 , piece_id=JCTZTTV4WTYQEXKOOUY6DHZNLUZT4MRU5JUPXTHVSSBOOCOHO4LA, seg_pos={0 0}, upload=494511]: Done uploading piece. err: <nil> Dec 29 03:30:06.186953075 (564h1m6.05803612s) [noise=true , address=145.79.227.43:4159 , piece_id=UGEOANOJN333PQUVQY3LXEFN7ZUIRNGCJPZKUMUALIU6QSHWHHNA, seg_pos={0 0}, upload=494511]: Done uploading piece. err: <nil> Dec 29 03:30:06.554325818 (564h1m6.425408853s) [noise=true , address=45.91.116.12:4227 , piece_id=URBYIJUXJGOAZGEZQ6C4IWG57SIOWOG2D7MXG5ZIJSRITG7HNLXA, seg_pos={0 0}, upload=494511]: Done uploading piece. err: <nil> Dec 29 03:30:06.554354872 (564h1m6.425437917s) [seg_pos={0 0}, upload=494511]: Segment reached optimal threshold of 13 pieces. Dec 29 03:30:06.554611562 (564h1m6.425694607s) [noise=true , address=145.79.227.50:4373 , piece_id=F446MJAV44FV5UWERNOAL3VGLBCQ7V2G3NJ3Y524YHX2U3EMPP5A, seg_pos={0 0}, upload=494511]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz): protocol: write tcp 188.166.225.170:41880->145.79.227.50:4373: use of closed network connection; write tcp 188.166.225.170:41880->145.79.227.50:4373: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41880->145.79.227.50:4373: use of closed network connection Dec 29 03:30:06.554622622 (564h1m6.425705668s) [noise=true , address=145.79.227.50:4373 , piece_id=F446MJAV44FV5UWERNOAL3VGLBCQ7V2G3NJ3Y524YHX2U3EMPP5A, seg_pos={0 0}, upload=494511]: Long tail cancellation due to optimal threshold Dec 29 03:30:06.554939796 (564h1m6.426022851s) [noise=true , address=77.90.31.12:4590 , piece_id=QSYQ4XVF2EUE52XL2QLGWEKGLRAN6EO2OAF5NGGLCJXEOA563OEQ, seg_pos={0 0}, upload=494511]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 29 03:30:06.554963231 (564h1m6.426046276s) [noise=true , address=77.90.31.12:4590 , piece_id=QSYQ4XVF2EUE52XL2QLGWEKGLRAN6EO2OAF5NGGLCJXEOA563OEQ, seg_pos={0 0}, upload=494511]: Long tail cancellation due to optimal threshold Dec 29 03:30:06.555230940 (564h1m6.426313985s) [noise=true , address=45.91.116.7:4010 , piece_id=CCM3YISUJC2OTCA7PFINNHNDBE64GTNEOE7J5A2VLRE5ZPURGCTQ, seg_pos={0 0}, upload=494511]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: context canceled; context canceled Dec 29 03:30:06.555239537 (564h1m6.426322583s) [noise=true , address=45.91.116.7:4010 , piece_id=CCM3YISUJC2OTCA7PFINNHNDBE64GTNEOE7J5A2VLRE5ZPURGCTQ, seg_pos={0 0}, upload=494511]: Long tail cancellation due to optimal threshold Dec 29 03:30:06.555257441 (564h1m6.426340486s) [seg_pos={0 0}, upload=494511]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:30:06.555341482 (564h1m6.426424527s) [upload=494511]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:30:06.555347493 (564h1m6.426430538s) [upload=494511]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:30:06.847283826 (564h1m6.718366871s) [upload=494511]: Done uploading segments... Dec 29 03:30:06.847307922 (564h1m6.718390967s) [upload=494511]: Upload finished. err: <nil> Duration: 30.003552623s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............................................XX............] duration: 4.162507325s trace: https://jaeger.dev.storj.io/trace/481fdf974580574a FAIL (2025-12-29 03:30:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.162535327s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............................................XX...........] duration: 3.673770189s trace: https://jaeger.dev.storj.io/trace/481fdf974580574a FAIL (2025-12-29 03:30:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.673840521s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 39.117853349s trace: https://jaeger.dev.storj.io/trace/1584175ba8dba7fe FAIL (2025-12-29 03:30:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ece... Dec 29 03:20:05.592697598 (563h51m5.463780643s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593249339 (563h51m5.464332384s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593687443 (563h51m5.464770488s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594173598 (563h51m5.465256633s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594639957 (563h51m5.465723002s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595331994 (563h51m5.466415039s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595834151 (563h51m5.466917206s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.596329844 (563h51m5.467412899s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599128643 (563h51m5.470211688s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599196953 (563h51m5.470279988s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.601094367 (563h51m5.472177412s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.602868896 (563h51m5.473951941s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.604661328 (563h51m5.475744374s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.606462138 (563h51m5.477545183s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.609580124 (563h51m5.480663169s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.611644807 (563h51m5.482727852s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:06.901416710 (563h51m6.772499755s) [upload=494362]: Next returned nil segment Dec 29 03:20:06.901434123 (563h51m6.772517168s) [upload=494362]: Waiting for error group managing segments... Dec 29 03:20:07.237383793 (563h51m7.108466848s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.439111607 (563h51m7.310194652s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.462301153 (563h51m7.333384198s) [noise=true , address=145.79.227.21:4498 , piece_id=RTK3PTWGWTIQSNCGREQB37OID2RGK37QWEKTDYFQKASGH6SZCURA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.479891718 (563h51m7.350974763s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.486062297 (563h51m7.357145342s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.578667930 (563h51m7.449750975s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.701241555 (563h51m7.5723246s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.745868671 (563h51m7.616951706s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.799936903 (563h51m7.671019948s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.430929511 (563h51m8.302012556s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.432106221 (563h51m8.303189256s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.447633778 (563h51m8.318716813s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.459472520 (563h51m8.330555555s) [noise=true , address=145.79.227.34:4916 , piece_id=4U67LINR5A4GR2NIUVMB2IPTT5DJQOYHM6HDYKZEICKF5JY53VFA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499215165 (563h51m8.37029821s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499245653 (563h51m8.370328698s) [seg_pos={0 0}, upload=494362]: Segment reached optimal threshold of 14 pieces. Dec 29 03:20:08.499544722 (563h51m8.370627797s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.499555243 (563h51m8.370638288s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.499833272 (563h51m8.370916327s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection Dec 29 03:20:08.499843291 (563h51m8.370926366s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500107964 (563h51m8.371191019s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.500166165 (563h51m8.37124921s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500868092 (563h51m8.371951147s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53): protocol: context canceled; context canceled Dec 29 03:20:08.500880466 (563h51m8.371963511s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500945178 (563h51m8.372028223s) [seg_pos={0 0}, upload=494362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:20:08.501128358 (563h51m8.372211413s) [upload=494362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:20:08.501156331 (563h51m8.372239376s) [upload=494362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:20:08.795508553 (563h51m8.666591598s) [upload=494362]: Done uploading segments... Dec 29 03:20:08.795532639 (563h51m8.666615684s) [upload=494362]: Upload finished. err: <nil> Duration: 39.117883466s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.X.X....X.X.XX..XX.....XX.X.....X.X..XXXX..XXX.XX...X.X.....] duration: 8.094760579s trace: https://jaeger.dev.storj.io/trace/fd304c9583ba31a FAIL (2025-12-29 03:29:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 03:25:10.777514077 (563h56m10.648597122s) [upload=494438]: Starting upload Dec 29 03:25:10.777595341 (563h56m10.648678386s) [upload=494438]: Done starting upload Dec 29 03:25:10.777947953 (563h56m10.649030999s) [upload=494438]: Uploading segments... Dec 29 03:25:10.777993450 (563h56m10.649076495s) [upload=494438]: Got next segment. Inline: false Dec 29 03:25:10.778044597 (563h56m10.649127642s) [upload=494438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:25:10.778049566 (563h56m10.649132601s) [upload=494438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:25:11.023293341 (563h56m10.894376377s) [seg_pos={0 0}, upload=494438]: Begin upload segment... Dec 29 03:25:11.023338908 (563h56m10.894421953s) [seg_pos={0 0}, upload=494438]: Done begin upload segment. Dec 29 03:25:11.023377561 (563h56m10.894460596s) [noise=true , address=77.90.31.15:4697 , piece_id=6NRWSCWKICUNGEPQ5NKKBJF3AABDMQDIASETPSNXWMBJ2U7GL64A, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.023435903 (563h56m10.894518938s) [noise=true , address=145.79.227.37:4567 , piece_id=6Y7UJKYJZEX3OC2TM37RRWBXAN4QANNWS5HL74VJJQQNC7UVB7SA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.024545235 (563h56m10.89562827s) [noise=true , address=145.79.227.41:4087 , piece_id=T6JZVKG3D45TBA2CI24Q3ZOPKNBZYRQNBUA3THA3PZWSIOX6H3QA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.024761807 (563h56m10.895844843s) [noise=true , address=145.79.227.39:4013 , piece_id=OTXRXVS666KLF74RYWBS6KLW3TUX7RJQLQDMIXIRN43WAS6QOMMQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.025191245 (563h56m10.89627428s) [noise=true , address=145.79.227.50:4375 , piece_id=VFZNMYHCBTCK7DOVEKR6RDGD3KO6K3KJ3RJHGQ4LUM3NHFBLVWYA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.025365376 (563h56m10.896448412s) [noise=true , address=77.90.31.12:4589 , piece_id=HWDLPR4SNVR7GLKQYNJXSEFF5A4CNWZNY32HUBCWVU4DLRHJALQQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.026271602 (563h56m10.897354637s) [noise=true , address=145.79.227.36:4517 , piece_id=KKBCMNEBUQJMOCKPSUQWA5KMCEM4RMLVIXNHLS2WNG22UEQ3TWWQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.026422379 (563h56m10.897505414s) [noise=true , address=145.79.227.19:4079 , piece_id=BNYQNAWKFS3CWRGFTU2WZMD7B3JWBXE6TE73CCOG3UMXCTYFVYDQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.027464674 (563h56m10.898547709s) [noise=true , address=145.79.227.11:4731 , piece_id=YR5EXBN5CMLGR65ABOQJCRPG2UTYNLJZSKPADO3DA45PUJYRAWSA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.027586847 (563h56m10.898669882s) [noise=true , address=45.91.116.14:4191 , piece_id=WBXLM4TNAT4SJX3UTU7T36A2IEGJVUHW57PNM3PCAO33UGMXRT4Q, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.028313140 (563h56m10.899396175s) [noise=true , address=145.79.227.48:4303 , piece_id=2UNUYZ3HHI4HLZXOVSCQXWCUAKPDXE5YLUVBCYEL4IVSMGE4WLVA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.028528690 (563h56m10.899611735s) [noise=true , address=145.79.227.51:4409 , piece_id=AUI6FSNO6YEH5JMSTELCTYBAQCSJCYW36CG56IKDYGT2MYJFFTOQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.028869249 (563h56m10.899952284s) [noise=true , address=45.91.116.5:4299 , piece_id=6KQ23R6NG74TJ472ENMR5C35AWNA5TAW53L3KFB4CZWER3KYTAMQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.029374130 (563h56m10.900457175s) [noise=true , address=77.90.31.11:4555 , piece_id=X43PH3ICCMDXVP73DUVGDLMRQJ3YPF5D4TQMRN5XPCDP55IMFZBQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.029760004 (563h56m10.900843049s) [noise=true , address=77.90.31.13:4627 , piece_id=2XWPNVP7JHGKBYD32EGUG5BECQH6YAXLLMVEKHQB62XDJFKXIJHQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.029960446 (563h56m10.901043491s) [noise=true , address=45.91.116.10:4084 , piece_id=JCIRURL7UZWCP7KQE3M57FXBDACKOGAKIYMT3LZH2C7WVCLZ7XNA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:12.720599483 (563h56m12.591682528s) [upload=494438]: Next returned nil segment Dec 29 03:25:12.720617647 (563h56m12.591700693s) [upload=494438]: Waiting for error group managing segments... Dec 29 03:25:13.004770289 (563h56m12.875853334s) [noise=true , address=145.79.227.37:4567 , piece_id=6Y7UJKYJZEX3OC2TM37RRWBXAN4QANNWS5HL74VJJQQNC7UVB7SA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.015495160 (563h56m12.886578205s) [noise=true , address=145.79.227.36:4517 , piece_id=KKBCMNEBUQJMOCKPSUQWA5KMCEM4RMLVIXNHLS2WNG22UEQ3TWWQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.016524962 (563h56m12.887608007s) [noise=true , address=145.79.227.39:4013 , piece_id=OTXRXVS666KLF74RYWBS6KLW3TUX7RJQLQDMIXIRN43WAS6QOMMQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.018057500 (563h56m12.889140545s) [noise=true , address=145.79.227.50:4375 , piece_id=VFZNMYHCBTCK7DOVEKR6RDGD3KO6K3KJ3RJHGQ4LUM3NHFBLVWYA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.019998305 (563h56m12.89108134s) [noise=true , address=145.79.227.41:4087 , piece_id=T6JZVKG3D45TBA2CI24Q3ZOPKNBZYRQNBUA3THA3PZWSIOX6H3QA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.027899229 (563h56m12.898982275s) [noise=true , address=145.79.227.51:4409 , piece_id=AUI6FSNO6YEH5JMSTELCTYBAQCSJCYW36CG56IKDYGT2MYJFFTOQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.030448854 (563h56m12.901531899s) [noise=true , address=145.79.227.11:4731 , piece_id=YR5EXBN5CMLGR65ABOQJCRPG2UTYNLJZSKPADO3DA45PUJYRAWSA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.039699990 (563h56m12.910783025s) [noise=true , address=145.79.227.48:4303 , piece_id=2UNUYZ3HHI4HLZXOVSCQXWCUAKPDXE5YLUVBCYEL4IVSMGE4WLVA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.043482302 (563h56m12.914565337s) [noise=true , address=145.79.227.19:4079 , piece_id=BNYQNAWKFS3CWRGFTU2WZMD7B3JWBXE6TE73CCOG3UMXCTYFVYDQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.201215966 (563h56m13.072299001s) [noise=true , address=77.90.31.11:4555 , piece_id=X43PH3ICCMDXVP73DUVGDLMRQJ3YPF5D4TQMRN5XPCDP55IMFZBQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.415979141 (563h56m13.287062186s) [noise=true , address=45.91.116.10:4084 , piece_id=JCIRURL7UZWCP7KQE3M57FXBDACKOGAKIYMT3LZH2C7WVCLZ7XNA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.423595834 (563h56m13.294678869s) [noise=true , address=45.91.116.5:4299 , piece_id=6KQ23R6NG74TJ472ENMR5C35AWNA5TAW53L3KFB4CZWER3KYTAMQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.937189360 (563h56m13.808272395s) [noise=true , address=77.90.31.15:4697 , piece_id=6NRWSCWKICUNGEPQ5NKKBJF3AABDMQDIASETPSNXWMBJ2U7GL64A, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.937208195 (563h56m13.80829124s) [seg_pos={0 0}, upload=494438]: Segment reached optimal threshold of 13 pieces. Dec 29 03:25:13.937375103 (563h56m13.808458138s) [noise=true , address=45.91.116.14:4191 , piece_id=WBXLM4TNAT4SJX3UTU7T36A2IEGJVUHW57PNM3PCAO33UGMXRT4Q, seg_pos={0 0}, upload=494438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 29 03:25:13.937381205 (563h56m13.80846425s) [noise=true , address=45.91.116.14:4191 , piece_id=WBXLM4TNAT4SJX3UTU7T36A2IEGJVUHW57PNM3PCAO33UGMXRT4Q, seg_pos={0 0}, upload=494438]: Long tail cancellation due to optimal threshold Dec 29 03:25:13.937552962 (563h56m13.808636007s) [noise=true , address=77.90.31.13:4627 , piece_id=2XWPNVP7JHGKBYD32EGUG5BECQH6YAXLLMVEKHQB62XDJFKXIJHQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:58080->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:58080->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58080->77.90.31.13:4627: use of closed network connection Dec 29 03:25:13.937561899 (563h56m13.808644944s) [noise=true , address=77.90.31.13:4627 , piece_id=2XWPNVP7JHGKBYD32EGUG5BECQH6YAXLLMVEKHQB62XDJFKXIJHQ, seg_pos={0 0}, upload=494438]: Long tail cancellation due to optimal threshold Dec 29 03:25:13.937591485 (563h56m13.80867452s) [noise=true , address=77.90.31.12:4589 , piece_id=HWDLPR4SNVR7GLKQYNJXSEFF5A4CNWZNY32HUBCWVU4DLRHJALQQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 29 03:25:13.937598710 (563h56m13.808681745s) [noise=true , address=77.90.31.12:4589 , piece_id=HWDLPR4SNVR7GLKQYNJXSEFF5A4CNWZNY32HUBCWVU4DLRHJALQQ, seg_pos={0 0}, upload=494438]: Long tail cancellation due to optimal threshold Dec 29 03:25:13.937618697 (563h56m13.808701732s) [seg_pos={0 0}, upload=494438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:25:13.937685905 (563h56m13.80876894s) [upload=494438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:25:13.937690594 (563h56m13.808773629s) [upload=494438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:25:14.227611579 (563h56m14.098694624s) [upload=494438]: Done uploading segments... Dec 29 03:25:14.227631487 (563h56m14.098714523s) [upload=494438]: Upload finished. err: <nil> Duration: 8.094789322s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 32.380608726s trace: https://jaeger.dev.storj.io/trace/1584175ba8dba7fe FAIL (2025-12-29 03:29:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ece... Dec 29 03:20:05.592697598 (563h51m5.463780643s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593249339 (563h51m5.464332384s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593687443 (563h51m5.464770488s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594173598 (563h51m5.465256633s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594639957 (563h51m5.465723002s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595331994 (563h51m5.466415039s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595834151 (563h51m5.466917206s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.596329844 (563h51m5.467412899s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599128643 (563h51m5.470211688s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599196953 (563h51m5.470279988s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.601094367 (563h51m5.472177412s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.602868896 (563h51m5.473951941s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.604661328 (563h51m5.475744374s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.606462138 (563h51m5.477545183s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.609580124 (563h51m5.480663169s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.611644807 (563h51m5.482727852s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:06.901416710 (563h51m6.772499755s) [upload=494362]: Next returned nil segment Dec 29 03:20:06.901434123 (563h51m6.772517168s) [upload=494362]: Waiting for error group managing segments... Dec 29 03:20:07.237383793 (563h51m7.108466848s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.439111607 (563h51m7.310194652s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.462301153 (563h51m7.333384198s) [noise=true , address=145.79.227.21:4498 , piece_id=RTK3PTWGWTIQSNCGREQB37OID2RGK37QWEKTDYFQKASGH6SZCURA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.479891718 (563h51m7.350974763s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.486062297 (563h51m7.357145342s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.578667930 (563h51m7.449750975s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.701241555 (563h51m7.5723246s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.745868671 (563h51m7.616951706s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.799936903 (563h51m7.671019948s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.430929511 (563h51m8.302012556s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.432106221 (563h51m8.303189256s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.447633778 (563h51m8.318716813s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.459472520 (563h51m8.330555555s) [noise=true , address=145.79.227.34:4916 , piece_id=4U67LINR5A4GR2NIUVMB2IPTT5DJQOYHM6HDYKZEICKF5JY53VFA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499215165 (563h51m8.37029821s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499245653 (563h51m8.370328698s) [seg_pos={0 0}, upload=494362]: Segment reached optimal threshold of 14 pieces. Dec 29 03:20:08.499544722 (563h51m8.370627797s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.499555243 (563h51m8.370638288s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.499833272 (563h51m8.370916327s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection Dec 29 03:20:08.499843291 (563h51m8.370926366s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500107964 (563h51m8.371191019s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.500166165 (563h51m8.37124921s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500868092 (563h51m8.371951147s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53): protocol: context canceled; context canceled Dec 29 03:20:08.500880466 (563h51m8.371963511s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500945178 (563h51m8.372028223s) [seg_pos={0 0}, upload=494362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:20:08.501128358 (563h51m8.372211413s) [upload=494362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:20:08.501156331 (563h51m8.372239376s) [upload=494362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:20:08.795508553 (563h51m8.666591598s) [upload=494362]: Done uploading segments... Dec 29 03:20:08.795532639 (563h51m8.666615684s) [upload=494362]: Upload finished. err: <nil> Duration: 32.380629977s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................................................XX..........] duration: 3.539130749s trace: https://jaeger.dev.storj.io/trace/481fdf974580574a FAIL (2025-12-29 03:29:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.539161757s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [X.X.X....X.X.XX..XX.....XX.X.....X.X..XXXX..XXX.XX...X.X....] duration: 6.245499181s trace: https://jaeger.dev.storj.io/trace/fd304c9583ba31a FAIL (2025-12-29 03:28:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 03:25:10.777514077 (563h56m10.648597122s) [upload=494438]: Starting upload Dec 29 03:25:10.777595341 (563h56m10.648678386s) [upload=494438]: Done starting upload Dec 29 03:25:10.777947953 (563h56m10.649030999s) [upload=494438]: Uploading segments... Dec 29 03:25:10.777993450 (563h56m10.649076495s) [upload=494438]: Got next segment. Inline: false Dec 29 03:25:10.778044597 (563h56m10.649127642s) [upload=494438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:25:10.778049566 (563h56m10.649132601s) [upload=494438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:25:11.023293341 (563h56m10.894376377s) [seg_pos={0 0}, upload=494438]: Begin upload segment... Dec 29 03:25:11.023338908 (563h56m10.894421953s) [seg_pos={0 0}, upload=494438]: Done begin upload segment. Dec 29 03:25:11.023377561 (563h56m10.894460596s) [noise=true , address=77.90.31.15:4697 , piece_id=6NRWSCWKICUNGEPQ5NKKBJF3AABDMQDIASETPSNXWMBJ2U7GL64A, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.023435903 (563h56m10.894518938s) [noise=true , address=145.79.227.37:4567 , piece_id=6Y7UJKYJZEX3OC2TM37RRWBXAN4QANNWS5HL74VJJQQNC7UVB7SA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.024545235 (563h56m10.89562827s) [noise=true , address=145.79.227.41:4087 , piece_id=T6JZVKG3D45TBA2CI24Q3ZOPKNBZYRQNBUA3THA3PZWSIOX6H3QA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.024761807 (563h56m10.895844843s) [noise=true , address=145.79.227.39:4013 , piece_id=OTXRXVS666KLF74RYWBS6KLW3TUX7RJQLQDMIXIRN43WAS6QOMMQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.025191245 (563h56m10.89627428s) [noise=true , address=145.79.227.50:4375 , piece_id=VFZNMYHCBTCK7DOVEKR6RDGD3KO6K3KJ3RJHGQ4LUM3NHFBLVWYA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.025365376 (563h56m10.896448412s) [noise=true , address=77.90.31.12:4589 , piece_id=HWDLPR4SNVR7GLKQYNJXSEFF5A4CNWZNY32HUBCWVU4DLRHJALQQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.026271602 (563h56m10.897354637s) [noise=true , address=145.79.227.36:4517 , piece_id=KKBCMNEBUQJMOCKPSUQWA5KMCEM4RMLVIXNHLS2WNG22UEQ3TWWQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.026422379 (563h56m10.897505414s) [noise=true , address=145.79.227.19:4079 , piece_id=BNYQNAWKFS3CWRGFTU2WZMD7B3JWBXE6TE73CCOG3UMXCTYFVYDQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.027464674 (563h56m10.898547709s) [noise=true , address=145.79.227.11:4731 , piece_id=YR5EXBN5CMLGR65ABOQJCRPG2UTYNLJZSKPADO3DA45PUJYRAWSA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.027586847 (563h56m10.898669882s) [noise=true , address=45.91.116.14:4191 , piece_id=WBXLM4TNAT4SJX3UTU7T36A2IEGJVUHW57PNM3PCAO33UGMXRT4Q, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.028313140 (563h56m10.899396175s) [noise=true , address=145.79.227.48:4303 , piece_id=2UNUYZ3HHI4HLZXOVSCQXWCUAKPDXE5YLUVBCYEL4IVSMGE4WLVA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.028528690 (563h56m10.899611735s) [noise=true , address=145.79.227.51:4409 , piece_id=AUI6FSNO6YEH5JMSTELCTYBAQCSJCYW36CG56IKDYGT2MYJFFTOQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.028869249 (563h56m10.899952284s) [noise=true , address=45.91.116.5:4299 , piece_id=6KQ23R6NG74TJ472ENMR5C35AWNA5TAW53L3KFB4CZWER3KYTAMQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.029374130 (563h56m10.900457175s) [noise=true , address=77.90.31.11:4555 , piece_id=X43PH3ICCMDXVP73DUVGDLMRQJ3YPF5D4TQMRN5XPCDP55IMFZBQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.029760004 (563h56m10.900843049s) [noise=true , address=77.90.31.13:4627 , piece_id=2XWPNVP7JHGKBYD32EGUG5BECQH6YAXLLMVEKHQB62XDJFKXIJHQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.029960446 (563h56m10.901043491s) [noise=true , address=45.91.116.10:4084 , piece_id=JCIRURL7UZWCP7KQE3M57FXBDACKOGAKIYMT3LZH2C7WVCLZ7XNA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:12.720599483 (563h56m12.591682528s) [upload=494438]: Next returned nil segment Dec 29 03:25:12.720617647 (563h56m12.591700693s) [upload=494438]: Waiting for error group managing segments... Dec 29 03:25:13.004770289 (563h56m12.875853334s) [noise=true , address=145.79.227.37:4567 , piece_id=6Y7UJKYJZEX3OC2TM37RRWBXAN4QANNWS5HL74VJJQQNC7UVB7SA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.015495160 (563h56m12.886578205s) [noise=true , address=145.79.227.36:4517 , piece_id=KKBCMNEBUQJMOCKPSUQWA5KMCEM4RMLVIXNHLS2WNG22UEQ3TWWQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.016524962 (563h56m12.887608007s) [noise=true , address=145.79.227.39:4013 , piece_id=OTXRXVS666KLF74RYWBS6KLW3TUX7RJQLQDMIXIRN43WAS6QOMMQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.018057500 (563h56m12.889140545s) [noise=true , address=145.79.227.50:4375 , piece_id=VFZNMYHCBTCK7DOVEKR6RDGD3KO6K3KJ3RJHGQ4LUM3NHFBLVWYA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.019998305 (563h56m12.89108134s) [noise=true , address=145.79.227.41:4087 , piece_id=T6JZVKG3D45TBA2CI24Q3ZOPKNBZYRQNBUA3THA3PZWSIOX6H3QA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.027899229 (563h56m12.898982275s) [noise=true , address=145.79.227.51:4409 , piece_id=AUI6FSNO6YEH5JMSTELCTYBAQCSJCYW36CG56IKDYGT2MYJFFTOQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.030448854 (563h56m12.901531899s) [noise=true , address=145.79.227.11:4731 , piece_id=YR5EXBN5CMLGR65ABOQJCRPG2UTYNLJZSKPADO3DA45PUJYRAWSA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.039699990 (563h56m12.910783025s) [noise=true , address=145.79.227.48:4303 , piece_id=2UNUYZ3HHI4HLZXOVSCQXWCUAKPDXE5YLUVBCYEL4IVSMGE4WLVA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.043482302 (563h56m12.914565337s) [noise=true , address=145.79.227.19:4079 , piece_id=BNYQNAWKFS3CWRGFTU2WZMD7B3JWBXE6TE73CCOG3UMXCTYFVYDQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.201215966 (563h56m13.072299001s) [noise=true , address=77.90.31.11:4555 , piece_id=X43PH3ICCMDXVP73DUVGDLMRQJ3YPF5D4TQMRN5XPCDP55IMFZBQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.415979141 (563h56m13.287062186s) [noise=true , address=45.91.116.10:4084 , piece_id=JCIRURL7UZWCP7KQE3M57FXBDACKOGAKIYMT3LZH2C7WVCLZ7XNA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.423595834 (563h56m13.294678869s) [noise=true , address=45.91.116.5:4299 , piece_id=6KQ23R6NG74TJ472ENMR5C35AWNA5TAW53L3KFB4CZWER3KYTAMQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.937189360 (563h56m13.808272395s) [noise=true , address=77.90.31.15:4697 , piece_id=6NRWSCWKICUNGEPQ5NKKBJF3AABDMQDIASETPSNXWMBJ2U7GL64A, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.937208195 (563h56m13.80829124s) [seg_pos={0 0}, upload=494438]: Segment reached optimal threshold of 13 pieces. Dec 29 03:25:13.937375103 (563h56m13.808458138s) [noise=true , address=45.91.116.14:4191 , piece_id=WBXLM4TNAT4SJX3UTU7T36A2IEGJVUHW57PNM3PCAO33UGMXRT4Q, seg_pos={0 0}, upload=494438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 29 03:25:13.937381205 (563h56m13.80846425s) [noise=true , address=45.91.116.14:4191 , piece_id=WBXLM4TNAT4SJX3UTU7T36A2IEGJVUHW57PNM3PCAO33UGMXRT4Q, seg_pos={0 0}, upload=494438]: Long tail cancellation due to optimal threshold Dec 29 03:25:13.937552962 (563h56m13.808636007s) [noise=true , address=77.90.31.13:4627 , piece_id=2XWPNVP7JHGKBYD32EGUG5BECQH6YAXLLMVEKHQB62XDJFKXIJHQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:58080->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:58080->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58080->77.90.31.13:4627: use of closed network connection Dec 29 03:25:13.937561899 (563h56m13.808644944s) [noise=true , address=77.90.31.13:4627 , piece_id=2XWPNVP7JHGKBYD32EGUG5BECQH6YAXLLMVEKHQB62XDJFKXIJHQ, seg_pos={0 0}, upload=494438]: Long tail cancellation due to optimal threshold Dec 29 03:25:13.937591485 (563h56m13.80867452s) [noise=true , address=77.90.31.12:4589 , piece_id=HWDLPR4SNVR7GLKQYNJXSEFF5A4CNWZNY32HUBCWVU4DLRHJALQQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 29 03:25:13.937598710 (563h56m13.808681745s) [noise=true , address=77.90.31.12:4589 , piece_id=HWDLPR4SNVR7GLKQYNJXSEFF5A4CNWZNY32HUBCWVU4DLRHJALQQ, seg_pos={0 0}, upload=494438]: Long tail cancellation due to optimal threshold Dec 29 03:25:13.937618697 (563h56m13.808701732s) [seg_pos={0 0}, upload=494438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:25:13.937685905 (563h56m13.80876894s) [upload=494438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:25:13.937690594 (563h56m13.808773629s) [upload=494438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:25:14.227611579 (563h56m14.098694624s) [upload=494438]: Done uploading segments... Dec 29 03:25:14.227631487 (563h56m14.098714523s) [upload=494438]: Upload finished. err: <nil> Duration: 6.245538026s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................................................XX.........] duration: 3.546031243s trace: https://jaeger.dev.storj.io/trace/481fdf974580574a FAIL (2025-12-29 03:28:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.546116355s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.................................................X.........] duration: 15.412544967s trace: https://jaeger.dev.storj.io/trace/1584175ba8dba7fe FAIL (2025-12-29 03:28:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ece... Dec 29 03:20:05.592697598 (563h51m5.463780643s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593249339 (563h51m5.464332384s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593687443 (563h51m5.464770488s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594173598 (563h51m5.465256633s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594639957 (563h51m5.465723002s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595331994 (563h51m5.466415039s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595834151 (563h51m5.466917206s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.596329844 (563h51m5.467412899s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599128643 (563h51m5.470211688s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599196953 (563h51m5.470279988s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.601094367 (563h51m5.472177412s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.602868896 (563h51m5.473951941s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.604661328 (563h51m5.475744374s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.606462138 (563h51m5.477545183s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.609580124 (563h51m5.480663169s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.611644807 (563h51m5.482727852s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:06.901416710 (563h51m6.772499755s) [upload=494362]: Next returned nil segment Dec 29 03:20:06.901434123 (563h51m6.772517168s) [upload=494362]: Waiting for error group managing segments... Dec 29 03:20:07.237383793 (563h51m7.108466848s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.439111607 (563h51m7.310194652s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.462301153 (563h51m7.333384198s) [noise=true , address=145.79.227.21:4498 , piece_id=RTK3PTWGWTIQSNCGREQB37OID2RGK37QWEKTDYFQKASGH6SZCURA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.479891718 (563h51m7.350974763s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.486062297 (563h51m7.357145342s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.578667930 (563h51m7.449750975s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.701241555 (563h51m7.5723246s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.745868671 (563h51m7.616951706s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.799936903 (563h51m7.671019948s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.430929511 (563h51m8.302012556s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.432106221 (563h51m8.303189256s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.447633778 (563h51m8.318716813s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.459472520 (563h51m8.330555555s) [noise=true , address=145.79.227.34:4916 , piece_id=4U67LINR5A4GR2NIUVMB2IPTT5DJQOYHM6HDYKZEICKF5JY53VFA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499215165 (563h51m8.37029821s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499245653 (563h51m8.370328698s) [seg_pos={0 0}, upload=494362]: Segment reached optimal threshold of 14 pieces. Dec 29 03:20:08.499544722 (563h51m8.370627797s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.499555243 (563h51m8.370638288s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.499833272 (563h51m8.370916327s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection Dec 29 03:20:08.499843291 (563h51m8.370926366s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500107964 (563h51m8.371191019s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.500166165 (563h51m8.37124921s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500868092 (563h51m8.371951147s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53): protocol: context canceled; context canceled Dec 29 03:20:08.500880466 (563h51m8.371963511s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500945178 (563h51m8.372028223s) [seg_pos={0 0}, upload=494362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:20:08.501128358 (563h51m8.372211413s) [upload=494362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:20:08.501156331 (563h51m8.372239376s) [upload=494362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:20:08.795508553 (563h51m8.666591598s) [upload=494362]: Done uploading segments... Dec 29 03:20:08.795532639 (563h51m8.666615684s) [upload=494362]: Upload finished. err: <nil> Duration: 15.412581067s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X.................................................X........] duration: 31.793915023s trace: https://jaeger.dev.storj.io/trace/1584175ba8dba7fe FAIL (2025-12-29 03:27:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ece... Dec 29 03:20:05.592697598 (563h51m5.463780643s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593249339 (563h51m5.464332384s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593687443 (563h51m5.464770488s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594173598 (563h51m5.465256633s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594639957 (563h51m5.465723002s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595331994 (563h51m5.466415039s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595834151 (563h51m5.466917206s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.596329844 (563h51m5.467412899s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599128643 (563h51m5.470211688s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599196953 (563h51m5.470279988s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.601094367 (563h51m5.472177412s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.602868896 (563h51m5.473951941s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.604661328 (563h51m5.475744374s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.606462138 (563h51m5.477545183s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.609580124 (563h51m5.480663169s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.611644807 (563h51m5.482727852s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:06.901416710 (563h51m6.772499755s) [upload=494362]: Next returned nil segment Dec 29 03:20:06.901434123 (563h51m6.772517168s) [upload=494362]: Waiting for error group managing segments... Dec 29 03:20:07.237383793 (563h51m7.108466848s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.439111607 (563h51m7.310194652s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.462301153 (563h51m7.333384198s) [noise=true , address=145.79.227.21:4498 , piece_id=RTK3PTWGWTIQSNCGREQB37OID2RGK37QWEKTDYFQKASGH6SZCURA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.479891718 (563h51m7.350974763s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.486062297 (563h51m7.357145342s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.578667930 (563h51m7.449750975s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.701241555 (563h51m7.5723246s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.745868671 (563h51m7.616951706s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.799936903 (563h51m7.671019948s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.430929511 (563h51m8.302012556s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.432106221 (563h51m8.303189256s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.447633778 (563h51m8.318716813s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.459472520 (563h51m8.330555555s) [noise=true , address=145.79.227.34:4916 , piece_id=4U67LINR5A4GR2NIUVMB2IPTT5DJQOYHM6HDYKZEICKF5JY53VFA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499215165 (563h51m8.37029821s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499245653 (563h51m8.370328698s) [seg_pos={0 0}, upload=494362]: Segment reached optimal threshold of 14 pieces. Dec 29 03:20:08.499544722 (563h51m8.370627797s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.499555243 (563h51m8.370638288s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.499833272 (563h51m8.370916327s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection Dec 29 03:20:08.499843291 (563h51m8.370926366s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500107964 (563h51m8.371191019s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.500166165 (563h51m8.37124921s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500868092 (563h51m8.371951147s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53): protocol: context canceled; context canceled Dec 29 03:20:08.500880466 (563h51m8.371963511s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500945178 (563h51m8.372028223s) [seg_pos={0 0}, upload=494362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:20:08.501128358 (563h51m8.372211413s) [upload=494362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:20:08.501156331 (563h51m8.372239376s) [upload=494362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:20:08.795508553 (563h51m8.666591598s) [upload=494362]: Done uploading segments... Dec 29 03:20:08.795532639 (563h51m8.666615684s) [upload=494362]: Upload finished. err: <nil> Duration: 31.793942816s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.X.X.X....X.X.XX..XX.....XX.X.....X.X..XXXX..XXX.XX...X.X...] duration: 5.883811085s trace: https://jaeger.dev.storj.io/trace/fd304c9583ba31a FAIL (2025-12-29 03:27:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 03:25:10.777514077 (563h56m10.648597122s) [upload=494438]: Starting upload Dec 29 03:25:10.777595341 (563h56m10.648678386s) [upload=494438]: Done starting upload Dec 29 03:25:10.777947953 (563h56m10.649030999s) [upload=494438]: Uploading segments... Dec 29 03:25:10.777993450 (563h56m10.649076495s) [upload=494438]: Got next segment. Inline: false Dec 29 03:25:10.778044597 (563h56m10.649127642s) [upload=494438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:25:10.778049566 (563h56m10.649132601s) [upload=494438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:25:11.023293341 (563h56m10.894376377s) [seg_pos={0 0}, upload=494438]: Begin upload segment... Dec 29 03:25:11.023338908 (563h56m10.894421953s) [seg_pos={0 0}, upload=494438]: Done begin upload segment. Dec 29 03:25:11.023377561 (563h56m10.894460596s) [noise=true , address=77.90.31.15:4697 , piece_id=6NRWSCWKICUNGEPQ5NKKBJF3AABDMQDIASETPSNXWMBJ2U7GL64A, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.023435903 (563h56m10.894518938s) [noise=true , address=145.79.227.37:4567 , piece_id=6Y7UJKYJZEX3OC2TM37RRWBXAN4QANNWS5HL74VJJQQNC7UVB7SA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.024545235 (563h56m10.89562827s) [noise=true , address=145.79.227.41:4087 , piece_id=T6JZVKG3D45TBA2CI24Q3ZOPKNBZYRQNBUA3THA3PZWSIOX6H3QA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.024761807 (563h56m10.895844843s) [noise=true , address=145.79.227.39:4013 , piece_id=OTXRXVS666KLF74RYWBS6KLW3TUX7RJQLQDMIXIRN43WAS6QOMMQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.025191245 (563h56m10.89627428s) [noise=true , address=145.79.227.50:4375 , piece_id=VFZNMYHCBTCK7DOVEKR6RDGD3KO6K3KJ3RJHGQ4LUM3NHFBLVWYA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.025365376 (563h56m10.896448412s) [noise=true , address=77.90.31.12:4589 , piece_id=HWDLPR4SNVR7GLKQYNJXSEFF5A4CNWZNY32HUBCWVU4DLRHJALQQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.026271602 (563h56m10.897354637s) [noise=true , address=145.79.227.36:4517 , piece_id=KKBCMNEBUQJMOCKPSUQWA5KMCEM4RMLVIXNHLS2WNG22UEQ3TWWQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.026422379 (563h56m10.897505414s) [noise=true , address=145.79.227.19:4079 , piece_id=BNYQNAWKFS3CWRGFTU2WZMD7B3JWBXE6TE73CCOG3UMXCTYFVYDQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.027464674 (563h56m10.898547709s) [noise=true , address=145.79.227.11:4731 , piece_id=YR5EXBN5CMLGR65ABOQJCRPG2UTYNLJZSKPADO3DA45PUJYRAWSA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.027586847 (563h56m10.898669882s) [noise=true , address=45.91.116.14:4191 , piece_id=WBXLM4TNAT4SJX3UTU7T36A2IEGJVUHW57PNM3PCAO33UGMXRT4Q, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.028313140 (563h56m10.899396175s) [noise=true , address=145.79.227.48:4303 , piece_id=2UNUYZ3HHI4HLZXOVSCQXWCUAKPDXE5YLUVBCYEL4IVSMGE4WLVA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.028528690 (563h56m10.899611735s) [noise=true , address=145.79.227.51:4409 , piece_id=AUI6FSNO6YEH5JMSTELCTYBAQCSJCYW36CG56IKDYGT2MYJFFTOQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.028869249 (563h56m10.899952284s) [noise=true , address=45.91.116.5:4299 , piece_id=6KQ23R6NG74TJ472ENMR5C35AWNA5TAW53L3KFB4CZWER3KYTAMQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.029374130 (563h56m10.900457175s) [noise=true , address=77.90.31.11:4555 , piece_id=X43PH3ICCMDXVP73DUVGDLMRQJ3YPF5D4TQMRN5XPCDP55IMFZBQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.029760004 (563h56m10.900843049s) [noise=true , address=77.90.31.13:4627 , piece_id=2XWPNVP7JHGKBYD32EGUG5BECQH6YAXLLMVEKHQB62XDJFKXIJHQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.029960446 (563h56m10.901043491s) [noise=true , address=45.91.116.10:4084 , piece_id=JCIRURL7UZWCP7KQE3M57FXBDACKOGAKIYMT3LZH2C7WVCLZ7XNA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:12.720599483 (563h56m12.591682528s) [upload=494438]: Next returned nil segment Dec 29 03:25:12.720617647 (563h56m12.591700693s) [upload=494438]: Waiting for error group managing segments... Dec 29 03:25:13.004770289 (563h56m12.875853334s) [noise=true , address=145.79.227.37:4567 , piece_id=6Y7UJKYJZEX3OC2TM37RRWBXAN4QANNWS5HL74VJJQQNC7UVB7SA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.015495160 (563h56m12.886578205s) [noise=true , address=145.79.227.36:4517 , piece_id=KKBCMNEBUQJMOCKPSUQWA5KMCEM4RMLVIXNHLS2WNG22UEQ3TWWQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.016524962 (563h56m12.887608007s) [noise=true , address=145.79.227.39:4013 , piece_id=OTXRXVS666KLF74RYWBS6KLW3TUX7RJQLQDMIXIRN43WAS6QOMMQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.018057500 (563h56m12.889140545s) [noise=true , address=145.79.227.50:4375 , piece_id=VFZNMYHCBTCK7DOVEKR6RDGD3KO6K3KJ3RJHGQ4LUM3NHFBLVWYA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.019998305 (563h56m12.89108134s) [noise=true , address=145.79.227.41:4087 , piece_id=T6JZVKG3D45TBA2CI24Q3ZOPKNBZYRQNBUA3THA3PZWSIOX6H3QA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.027899229 (563h56m12.898982275s) [noise=true , address=145.79.227.51:4409 , piece_id=AUI6FSNO6YEH5JMSTELCTYBAQCSJCYW36CG56IKDYGT2MYJFFTOQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.030448854 (563h56m12.901531899s) [noise=true , address=145.79.227.11:4731 , piece_id=YR5EXBN5CMLGR65ABOQJCRPG2UTYNLJZSKPADO3DA45PUJYRAWSA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.039699990 (563h56m12.910783025s) [noise=true , address=145.79.227.48:4303 , piece_id=2UNUYZ3HHI4HLZXOVSCQXWCUAKPDXE5YLUVBCYEL4IVSMGE4WLVA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.043482302 (563h56m12.914565337s) [noise=true , address=145.79.227.19:4079 , piece_id=BNYQNAWKFS3CWRGFTU2WZMD7B3JWBXE6TE73CCOG3UMXCTYFVYDQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.201215966 (563h56m13.072299001s) [noise=true , address=77.90.31.11:4555 , piece_id=X43PH3ICCMDXVP73DUVGDLMRQJ3YPF5D4TQMRN5XPCDP55IMFZBQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.415979141 (563h56m13.287062186s) [noise=true , address=45.91.116.10:4084 , piece_id=JCIRURL7UZWCP7KQE3M57FXBDACKOGAKIYMT3LZH2C7WVCLZ7XNA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.423595834 (563h56m13.294678869s) [noise=true , address=45.91.116.5:4299 , piece_id=6KQ23R6NG74TJ472ENMR5C35AWNA5TAW53L3KFB4CZWER3KYTAMQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.937189360 (563h56m13.808272395s) [noise=true , address=77.90.31.15:4697 , piece_id=6NRWSCWKICUNGEPQ5NKKBJF3AABDMQDIASETPSNXWMBJ2U7GL64A, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.937208195 (563h56m13.80829124s) [seg_pos={0 0}, upload=494438]: Segment reached optimal threshold of 13 pieces. Dec 29 03:25:13.937375103 (563h56m13.808458138s) [noise=true , address=45.91.116.14:4191 , piece_id=WBXLM4TNAT4SJX3UTU7T36A2IEGJVUHW57PNM3PCAO33UGMXRT4Q, seg_pos={0 0}, upload=494438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 29 03:25:13.937381205 (563h56m13.80846425s) [noise=true , address=45.91.116.14:4191 , piece_id=WBXLM4TNAT4SJX3UTU7T36A2IEGJVUHW57PNM3PCAO33UGMXRT4Q, seg_pos={0 0}, upload=494438]: Long tail cancellation due to optimal threshold Dec 29 03:25:13.937552962 (563h56m13.808636007s) [noise=true , address=77.90.31.13:4627 , piece_id=2XWPNVP7JHGKBYD32EGUG5BECQH6YAXLLMVEKHQB62XDJFKXIJHQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:58080->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:58080->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58080->77.90.31.13:4627: use of closed network connection Dec 29 03:25:13.937561899 (563h56m13.808644944s) [noise=true , address=77.90.31.13:4627 , piece_id=2XWPNVP7JHGKBYD32EGUG5BECQH6YAXLLMVEKHQB62XDJFKXIJHQ, seg_pos={0 0}, upload=494438]: Long tail cancellation due to optimal threshold Dec 29 03:25:13.937591485 (563h56m13.80867452s) [noise=true , address=77.90.31.12:4589 , piece_id=HWDLPR4SNVR7GLKQYNJXSEFF5A4CNWZNY32HUBCWVU4DLRHJALQQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 29 03:25:13.937598710 (563h56m13.808681745s) [noise=true , address=77.90.31.12:4589 , piece_id=HWDLPR4SNVR7GLKQYNJXSEFF5A4CNWZNY32HUBCWVU4DLRHJALQQ, seg_pos={0 0}, upload=494438]: Long tail cancellation due to optimal threshold Dec 29 03:25:13.937618697 (563h56m13.808701732s) [seg_pos={0 0}, upload=494438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:25:13.937685905 (563h56m13.80876894s) [upload=494438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:25:13.937690594 (563h56m13.808773629s) [upload=494438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:25:14.227611579 (563h56m14.098694624s) [upload=494438]: Done uploading segments... Dec 29 03:25:14.227631487 (563h56m14.098714523s) [upload=494438]: Upload finished. err: <nil> Duration: 5.883840342s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................................................XX........] duration: 3.569191338s trace: https://jaeger.dev.storj.io/trace/481fdf974580574a FAIL (2025-12-29 03:27:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.569214933s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [..X.X.X....X.X.XX..XX.....XX.X.....X.X..XXXX..XXX.XX...X.X..] duration: 6.030576761s trace: https://jaeger.dev.storj.io/trace/fd304c9583ba31a FAIL (2025-12-29 03:26:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 03:25:10.777514077 (563h56m10.648597122s) [upload=494438]: Starting upload Dec 29 03:25:10.777595341 (563h56m10.648678386s) [upload=494438]: Done starting upload Dec 29 03:25:10.777947953 (563h56m10.649030999s) [upload=494438]: Uploading segments... Dec 29 03:25:10.777993450 (563h56m10.649076495s) [upload=494438]: Got next segment. Inline: false Dec 29 03:25:10.778044597 (563h56m10.649127642s) [upload=494438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:25:10.778049566 (563h56m10.649132601s) [upload=494438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:25:11.023293341 (563h56m10.894376377s) [seg_pos={0 0}, upload=494438]: Begin upload segment... Dec 29 03:25:11.023338908 (563h56m10.894421953s) [seg_pos={0 0}, upload=494438]: Done begin upload segment. Dec 29 03:25:11.023377561 (563h56m10.894460596s) [noise=true , address=77.90.31.15:4697 , piece_id=6NRWSCWKICUNGEPQ5NKKBJF3AABDMQDIASETPSNXWMBJ2U7GL64A, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.023435903 (563h56m10.894518938s) [noise=true , address=145.79.227.37:4567 , piece_id=6Y7UJKYJZEX3OC2TM37RRWBXAN4QANNWS5HL74VJJQQNC7UVB7SA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.024545235 (563h56m10.89562827s) [noise=true , address=145.79.227.41:4087 , piece_id=T6JZVKG3D45TBA2CI24Q3ZOPKNBZYRQNBUA3THA3PZWSIOX6H3QA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.024761807 (563h56m10.895844843s) [noise=true , address=145.79.227.39:4013 , piece_id=OTXRXVS666KLF74RYWBS6KLW3TUX7RJQLQDMIXIRN43WAS6QOMMQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.025191245 (563h56m10.89627428s) [noise=true , address=145.79.227.50:4375 , piece_id=VFZNMYHCBTCK7DOVEKR6RDGD3KO6K3KJ3RJHGQ4LUM3NHFBLVWYA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.025365376 (563h56m10.896448412s) [noise=true , address=77.90.31.12:4589 , piece_id=HWDLPR4SNVR7GLKQYNJXSEFF5A4CNWZNY32HUBCWVU4DLRHJALQQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.026271602 (563h56m10.897354637s) [noise=true , address=145.79.227.36:4517 , piece_id=KKBCMNEBUQJMOCKPSUQWA5KMCEM4RMLVIXNHLS2WNG22UEQ3TWWQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.026422379 (563h56m10.897505414s) [noise=true , address=145.79.227.19:4079 , piece_id=BNYQNAWKFS3CWRGFTU2WZMD7B3JWBXE6TE73CCOG3UMXCTYFVYDQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.027464674 (563h56m10.898547709s) [noise=true , address=145.79.227.11:4731 , piece_id=YR5EXBN5CMLGR65ABOQJCRPG2UTYNLJZSKPADO3DA45PUJYRAWSA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.027586847 (563h56m10.898669882s) [noise=true , address=45.91.116.14:4191 , piece_id=WBXLM4TNAT4SJX3UTU7T36A2IEGJVUHW57PNM3PCAO33UGMXRT4Q, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.028313140 (563h56m10.899396175s) [noise=true , address=145.79.227.48:4303 , piece_id=2UNUYZ3HHI4HLZXOVSCQXWCUAKPDXE5YLUVBCYEL4IVSMGE4WLVA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.028528690 (563h56m10.899611735s) [noise=true , address=145.79.227.51:4409 , piece_id=AUI6FSNO6YEH5JMSTELCTYBAQCSJCYW36CG56IKDYGT2MYJFFTOQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.028869249 (563h56m10.899952284s) [noise=true , address=45.91.116.5:4299 , piece_id=6KQ23R6NG74TJ472ENMR5C35AWNA5TAW53L3KFB4CZWER3KYTAMQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.029374130 (563h56m10.900457175s) [noise=true , address=77.90.31.11:4555 , piece_id=X43PH3ICCMDXVP73DUVGDLMRQJ3YPF5D4TQMRN5XPCDP55IMFZBQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.029760004 (563h56m10.900843049s) [noise=true , address=77.90.31.13:4627 , piece_id=2XWPNVP7JHGKBYD32EGUG5BECQH6YAXLLMVEKHQB62XDJFKXIJHQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.029960446 (563h56m10.901043491s) [noise=true , address=45.91.116.10:4084 , piece_id=JCIRURL7UZWCP7KQE3M57FXBDACKOGAKIYMT3LZH2C7WVCLZ7XNA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:12.720599483 (563h56m12.591682528s) [upload=494438]: Next returned nil segment Dec 29 03:25:12.720617647 (563h56m12.591700693s) [upload=494438]: Waiting for error group managing segments... Dec 29 03:25:13.004770289 (563h56m12.875853334s) [noise=true , address=145.79.227.37:4567 , piece_id=6Y7UJKYJZEX3OC2TM37RRWBXAN4QANNWS5HL74VJJQQNC7UVB7SA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.015495160 (563h56m12.886578205s) [noise=true , address=145.79.227.36:4517 , piece_id=KKBCMNEBUQJMOCKPSUQWA5KMCEM4RMLVIXNHLS2WNG22UEQ3TWWQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.016524962 (563h56m12.887608007s) [noise=true , address=145.79.227.39:4013 , piece_id=OTXRXVS666KLF74RYWBS6KLW3TUX7RJQLQDMIXIRN43WAS6QOMMQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.018057500 (563h56m12.889140545s) [noise=true , address=145.79.227.50:4375 , piece_id=VFZNMYHCBTCK7DOVEKR6RDGD3KO6K3KJ3RJHGQ4LUM3NHFBLVWYA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.019998305 (563h56m12.89108134s) [noise=true , address=145.79.227.41:4087 , piece_id=T6JZVKG3D45TBA2CI24Q3ZOPKNBZYRQNBUA3THA3PZWSIOX6H3QA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.027899229 (563h56m12.898982275s) [noise=true , address=145.79.227.51:4409 , piece_id=AUI6FSNO6YEH5JMSTELCTYBAQCSJCYW36CG56IKDYGT2MYJFFTOQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.030448854 (563h56m12.901531899s) [noise=true , address=145.79.227.11:4731 , piece_id=YR5EXBN5CMLGR65ABOQJCRPG2UTYNLJZSKPADO3DA45PUJYRAWSA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.039699990 (563h56m12.910783025s) [noise=true , address=145.79.227.48:4303 , piece_id=2UNUYZ3HHI4HLZXOVSCQXWCUAKPDXE5YLUVBCYEL4IVSMGE4WLVA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.043482302 (563h56m12.914565337s) [noise=true , address=145.79.227.19:4079 , piece_id=BNYQNAWKFS3CWRGFTU2WZMD7B3JWBXE6TE73CCOG3UMXCTYFVYDQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.201215966 (563h56m13.072299001s) [noise=true , address=77.90.31.11:4555 , piece_id=X43PH3ICCMDXVP73DUVGDLMRQJ3YPF5D4TQMRN5XPCDP55IMFZBQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.415979141 (563h56m13.287062186s) [noise=true , address=45.91.116.10:4084 , piece_id=JCIRURL7UZWCP7KQE3M57FXBDACKOGAKIYMT3LZH2C7WVCLZ7XNA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.423595834 (563h56m13.294678869s) [noise=true , address=45.91.116.5:4299 , piece_id=6KQ23R6NG74TJ472ENMR5C35AWNA5TAW53L3KFB4CZWER3KYTAMQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.937189360 (563h56m13.808272395s) [noise=true , address=77.90.31.15:4697 , piece_id=6NRWSCWKICUNGEPQ5NKKBJF3AABDMQDIASETPSNXWMBJ2U7GL64A, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.937208195 (563h56m13.80829124s) [seg_pos={0 0}, upload=494438]: Segment reached optimal threshold of 13 pieces. Dec 29 03:25:13.937375103 (563h56m13.808458138s) [noise=true , address=45.91.116.14:4191 , piece_id=WBXLM4TNAT4SJX3UTU7T36A2IEGJVUHW57PNM3PCAO33UGMXRT4Q, seg_pos={0 0}, upload=494438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 29 03:25:13.937381205 (563h56m13.80846425s) [noise=true , address=45.91.116.14:4191 , piece_id=WBXLM4TNAT4SJX3UTU7T36A2IEGJVUHW57PNM3PCAO33UGMXRT4Q, seg_pos={0 0}, upload=494438]: Long tail cancellation due to optimal threshold Dec 29 03:25:13.937552962 (563h56m13.808636007s) [noise=true , address=77.90.31.13:4627 , piece_id=2XWPNVP7JHGKBYD32EGUG5BECQH6YAXLLMVEKHQB62XDJFKXIJHQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:58080->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:58080->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58080->77.90.31.13:4627: use of closed network connection Dec 29 03:25:13.937561899 (563h56m13.808644944s) [noise=true , address=77.90.31.13:4627 , piece_id=2XWPNVP7JHGKBYD32EGUG5BECQH6YAXLLMVEKHQB62XDJFKXIJHQ, seg_pos={0 0}, upload=494438]: Long tail cancellation due to optimal threshold Dec 29 03:25:13.937591485 (563h56m13.80867452s) [noise=true , address=77.90.31.12:4589 , piece_id=HWDLPR4SNVR7GLKQYNJXSEFF5A4CNWZNY32HUBCWVU4DLRHJALQQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 29 03:25:13.937598710 (563h56m13.808681745s) [noise=true , address=77.90.31.12:4589 , piece_id=HWDLPR4SNVR7GLKQYNJXSEFF5A4CNWZNY32HUBCWVU4DLRHJALQQ, seg_pos={0 0}, upload=494438]: Long tail cancellation due to optimal threshold Dec 29 03:25:13.937618697 (563h56m13.808701732s) [seg_pos={0 0}, upload=494438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:25:13.937685905 (563h56m13.80876894s) [upload=494438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:25:13.937690594 (563h56m13.808773629s) [upload=494438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:25:14.227611579 (563h56m14.098694624s) [upload=494438]: Done uploading segments... Dec 29 03:25:14.227631487 (563h56m14.098714523s) [upload=494438]: Upload finished. err: <nil> Duration: 6.030603412s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...................................................XX.......] duration: 3.417075344s trace: https://jaeger.dev.storj.io/trace/481fdf974580574a FAIL (2025-12-29 03:26:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.417157018s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X.................................................X.......] duration: 32.779777834s trace: https://jaeger.dev.storj.io/trace/1584175ba8dba7fe FAIL (2025-12-29 03:26:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ece... Dec 29 03:20:05.592697598 (563h51m5.463780643s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593249339 (563h51m5.464332384s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593687443 (563h51m5.464770488s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594173598 (563h51m5.465256633s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594639957 (563h51m5.465723002s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595331994 (563h51m5.466415039s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595834151 (563h51m5.466917206s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.596329844 (563h51m5.467412899s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599128643 (563h51m5.470211688s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599196953 (563h51m5.470279988s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.601094367 (563h51m5.472177412s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.602868896 (563h51m5.473951941s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.604661328 (563h51m5.475744374s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.606462138 (563h51m5.477545183s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.609580124 (563h51m5.480663169s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.611644807 (563h51m5.482727852s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:06.901416710 (563h51m6.772499755s) [upload=494362]: Next returned nil segment Dec 29 03:20:06.901434123 (563h51m6.772517168s) [upload=494362]: Waiting for error group managing segments... Dec 29 03:20:07.237383793 (563h51m7.108466848s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.439111607 (563h51m7.310194652s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.462301153 (563h51m7.333384198s) [noise=true , address=145.79.227.21:4498 , piece_id=RTK3PTWGWTIQSNCGREQB37OID2RGK37QWEKTDYFQKASGH6SZCURA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.479891718 (563h51m7.350974763s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.486062297 (563h51m7.357145342s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.578667930 (563h51m7.449750975s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.701241555 (563h51m7.5723246s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.745868671 (563h51m7.616951706s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.799936903 (563h51m7.671019948s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.430929511 (563h51m8.302012556s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.432106221 (563h51m8.303189256s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.447633778 (563h51m8.318716813s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.459472520 (563h51m8.330555555s) [noise=true , address=145.79.227.34:4916 , piece_id=4U67LINR5A4GR2NIUVMB2IPTT5DJQOYHM6HDYKZEICKF5JY53VFA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499215165 (563h51m8.37029821s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499245653 (563h51m8.370328698s) [seg_pos={0 0}, upload=494362]: Segment reached optimal threshold of 14 pieces. Dec 29 03:20:08.499544722 (563h51m8.370627797s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.499555243 (563h51m8.370638288s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.499833272 (563h51m8.370916327s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection Dec 29 03:20:08.499843291 (563h51m8.370926366s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500107964 (563h51m8.371191019s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.500166165 (563h51m8.37124921s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500868092 (563h51m8.371951147s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53): protocol: context canceled; context canceled Dec 29 03:20:08.500880466 (563h51m8.371963511s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500945178 (563h51m8.372028223s) [seg_pos={0 0}, upload=494362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:20:08.501128358 (563h51m8.372211413s) [upload=494362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:20:08.501156331 (563h51m8.372239376s) [upload=494362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:20:08.795508553 (563h51m8.666591598s) [upload=494362]: Done uploading segments... Dec 29 03:20:08.795532639 (563h51m8.666615684s) [upload=494362]: Upload finished. err: <nil> Duration: 32.779859088s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X.................................................X......] duration: 36.013650063s trace: https://jaeger.dev.storj.io/trace/1584175ba8dba7fe FAIL (2025-12-29 03:25:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ece... Dec 29 03:20:05.592697598 (563h51m5.463780643s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593249339 (563h51m5.464332384s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593687443 (563h51m5.464770488s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594173598 (563h51m5.465256633s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594639957 (563h51m5.465723002s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595331994 (563h51m5.466415039s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595834151 (563h51m5.466917206s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.596329844 (563h51m5.467412899s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599128643 (563h51m5.470211688s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599196953 (563h51m5.470279988s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.601094367 (563h51m5.472177412s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.602868896 (563h51m5.473951941s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.604661328 (563h51m5.475744374s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.606462138 (563h51m5.477545183s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.609580124 (563h51m5.480663169s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.611644807 (563h51m5.482727852s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:06.901416710 (563h51m6.772499755s) [upload=494362]: Next returned nil segment Dec 29 03:20:06.901434123 (563h51m6.772517168s) [upload=494362]: Waiting for error group managing segments... Dec 29 03:20:07.237383793 (563h51m7.108466848s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.439111607 (563h51m7.310194652s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.462301153 (563h51m7.333384198s) [noise=true , address=145.79.227.21:4498 , piece_id=RTK3PTWGWTIQSNCGREQB37OID2RGK37QWEKTDYFQKASGH6SZCURA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.479891718 (563h51m7.350974763s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.486062297 (563h51m7.357145342s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.578667930 (563h51m7.449750975s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.701241555 (563h51m7.5723246s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.745868671 (563h51m7.616951706s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.799936903 (563h51m7.671019948s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.430929511 (563h51m8.302012556s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.432106221 (563h51m8.303189256s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.447633778 (563h51m8.318716813s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.459472520 (563h51m8.330555555s) [noise=true , address=145.79.227.34:4916 , piece_id=4U67LINR5A4GR2NIUVMB2IPTT5DJQOYHM6HDYKZEICKF5JY53VFA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499215165 (563h51m8.37029821s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499245653 (563h51m8.370328698s) [seg_pos={0 0}, upload=494362]: Segment reached optimal threshold of 14 pieces. Dec 29 03:20:08.499544722 (563h51m8.370627797s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.499555243 (563h51m8.370638288s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.499833272 (563h51m8.370916327s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection Dec 29 03:20:08.499843291 (563h51m8.370926366s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500107964 (563h51m8.371191019s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.500166165 (563h51m8.37124921s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500868092 (563h51m8.371951147s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53): protocol: context canceled; context canceled Dec 29 03:20:08.500880466 (563h51m8.371963511s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500945178 (563h51m8.372028223s) [seg_pos={0 0}, upload=494362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:20:08.501128358 (563h51m8.372211413s) [upload=494362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:20:08.501156331 (563h51m8.372239376s) [upload=494362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:20:08.795508553 (563h51m8.666591598s) [upload=494362]: Done uploading segments... Dec 29 03:20:08.795532639 (563h51m8.666615684s) [upload=494362]: Upload finished. err: <nil> Duration: 36.013673717s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [X..X.X.X....X.X.XX..XX.....XX.X.....X.X..XXXX..XXX.XX...X.X.] duration: 5.925510873s trace: https://jaeger.dev.storj.io/trace/fd304c9583ba31a FAIL (2025-12-29 03:25:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 03:25:10.777514077 (563h56m10.648597122s) [upload=494438]: Starting upload Dec 29 03:25:10.777595341 (563h56m10.648678386s) [upload=494438]: Done starting upload Dec 29 03:25:10.777947953 (563h56m10.649030999s) [upload=494438]: Uploading segments... Dec 29 03:25:10.777993450 (563h56m10.649076495s) [upload=494438]: Got next segment. Inline: false Dec 29 03:25:10.778044597 (563h56m10.649127642s) [upload=494438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:25:10.778049566 (563h56m10.649132601s) [upload=494438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:25:11.023293341 (563h56m10.894376377s) [seg_pos={0 0}, upload=494438]: Begin upload segment... Dec 29 03:25:11.023338908 (563h56m10.894421953s) [seg_pos={0 0}, upload=494438]: Done begin upload segment. Dec 29 03:25:11.023377561 (563h56m10.894460596s) [noise=true , address=77.90.31.15:4697 , piece_id=6NRWSCWKICUNGEPQ5NKKBJF3AABDMQDIASETPSNXWMBJ2U7GL64A, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.023435903 (563h56m10.894518938s) [noise=true , address=145.79.227.37:4567 , piece_id=6Y7UJKYJZEX3OC2TM37RRWBXAN4QANNWS5HL74VJJQQNC7UVB7SA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.024545235 (563h56m10.89562827s) [noise=true , address=145.79.227.41:4087 , piece_id=T6JZVKG3D45TBA2CI24Q3ZOPKNBZYRQNBUA3THA3PZWSIOX6H3QA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.024761807 (563h56m10.895844843s) [noise=true , address=145.79.227.39:4013 , piece_id=OTXRXVS666KLF74RYWBS6KLW3TUX7RJQLQDMIXIRN43WAS6QOMMQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.025191245 (563h56m10.89627428s) [noise=true , address=145.79.227.50:4375 , piece_id=VFZNMYHCBTCK7DOVEKR6RDGD3KO6K3KJ3RJHGQ4LUM3NHFBLVWYA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.025365376 (563h56m10.896448412s) [noise=true , address=77.90.31.12:4589 , piece_id=HWDLPR4SNVR7GLKQYNJXSEFF5A4CNWZNY32HUBCWVU4DLRHJALQQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.026271602 (563h56m10.897354637s) [noise=true , address=145.79.227.36:4517 , piece_id=KKBCMNEBUQJMOCKPSUQWA5KMCEM4RMLVIXNHLS2WNG22UEQ3TWWQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.026422379 (563h56m10.897505414s) [noise=true , address=145.79.227.19:4079 , piece_id=BNYQNAWKFS3CWRGFTU2WZMD7B3JWBXE6TE73CCOG3UMXCTYFVYDQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.027464674 (563h56m10.898547709s) [noise=true , address=145.79.227.11:4731 , piece_id=YR5EXBN5CMLGR65ABOQJCRPG2UTYNLJZSKPADO3DA45PUJYRAWSA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.027586847 (563h56m10.898669882s) [noise=true , address=45.91.116.14:4191 , piece_id=WBXLM4TNAT4SJX3UTU7T36A2IEGJVUHW57PNM3PCAO33UGMXRT4Q, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.028313140 (563h56m10.899396175s) [noise=true , address=145.79.227.48:4303 , piece_id=2UNUYZ3HHI4HLZXOVSCQXWCUAKPDXE5YLUVBCYEL4IVSMGE4WLVA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.028528690 (563h56m10.899611735s) [noise=true , address=145.79.227.51:4409 , piece_id=AUI6FSNO6YEH5JMSTELCTYBAQCSJCYW36CG56IKDYGT2MYJFFTOQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.028869249 (563h56m10.899952284s) [noise=true , address=45.91.116.5:4299 , piece_id=6KQ23R6NG74TJ472ENMR5C35AWNA5TAW53L3KFB4CZWER3KYTAMQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.029374130 (563h56m10.900457175s) [noise=true , address=77.90.31.11:4555 , piece_id=X43PH3ICCMDXVP73DUVGDLMRQJ3YPF5D4TQMRN5XPCDP55IMFZBQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.029760004 (563h56m10.900843049s) [noise=true , address=77.90.31.13:4627 , piece_id=2XWPNVP7JHGKBYD32EGUG5BECQH6YAXLLMVEKHQB62XDJFKXIJHQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.029960446 (563h56m10.901043491s) [noise=true , address=45.91.116.10:4084 , piece_id=JCIRURL7UZWCP7KQE3M57FXBDACKOGAKIYMT3LZH2C7WVCLZ7XNA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:12.720599483 (563h56m12.591682528s) [upload=494438]: Next returned nil segment Dec 29 03:25:12.720617647 (563h56m12.591700693s) [upload=494438]: Waiting for error group managing segments... Dec 29 03:25:13.004770289 (563h56m12.875853334s) [noise=true , address=145.79.227.37:4567 , piece_id=6Y7UJKYJZEX3OC2TM37RRWBXAN4QANNWS5HL74VJJQQNC7UVB7SA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.015495160 (563h56m12.886578205s) [noise=true , address=145.79.227.36:4517 , piece_id=KKBCMNEBUQJMOCKPSUQWA5KMCEM4RMLVIXNHLS2WNG22UEQ3TWWQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.016524962 (563h56m12.887608007s) [noise=true , address=145.79.227.39:4013 , piece_id=OTXRXVS666KLF74RYWBS6KLW3TUX7RJQLQDMIXIRN43WAS6QOMMQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.018057500 (563h56m12.889140545s) [noise=true , address=145.79.227.50:4375 , piece_id=VFZNMYHCBTCK7DOVEKR6RDGD3KO6K3KJ3RJHGQ4LUM3NHFBLVWYA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.019998305 (563h56m12.89108134s) [noise=true , address=145.79.227.41:4087 , piece_id=T6JZVKG3D45TBA2CI24Q3ZOPKNBZYRQNBUA3THA3PZWSIOX6H3QA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.027899229 (563h56m12.898982275s) [noise=true , address=145.79.227.51:4409 , piece_id=AUI6FSNO6YEH5JMSTELCTYBAQCSJCYW36CG56IKDYGT2MYJFFTOQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.030448854 (563h56m12.901531899s) [noise=true , address=145.79.227.11:4731 , piece_id=YR5EXBN5CMLGR65ABOQJCRPG2UTYNLJZSKPADO3DA45PUJYRAWSA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.039699990 (563h56m12.910783025s) [noise=true , address=145.79.227.48:4303 , piece_id=2UNUYZ3HHI4HLZXOVSCQXWCUAKPDXE5YLUVBCYEL4IVSMGE4WLVA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.043482302 (563h56m12.914565337s) [noise=true , address=145.79.227.19:4079 , piece_id=BNYQNAWKFS3CWRGFTU2WZMD7B3JWBXE6TE73CCOG3UMXCTYFVYDQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.201215966 (563h56m13.072299001s) [noise=true , address=77.90.31.11:4555 , piece_id=X43PH3ICCMDXVP73DUVGDLMRQJ3YPF5D4TQMRN5XPCDP55IMFZBQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.415979141 (563h56m13.287062186s) [noise=true , address=45.91.116.10:4084 , piece_id=JCIRURL7UZWCP7KQE3M57FXBDACKOGAKIYMT3LZH2C7WVCLZ7XNA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.423595834 (563h56m13.294678869s) [noise=true , address=45.91.116.5:4299 , piece_id=6KQ23R6NG74TJ472ENMR5C35AWNA5TAW53L3KFB4CZWER3KYTAMQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.937189360 (563h56m13.808272395s) [noise=true , address=77.90.31.15:4697 , piece_id=6NRWSCWKICUNGEPQ5NKKBJF3AABDMQDIASETPSNXWMBJ2U7GL64A, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.937208195 (563h56m13.80829124s) [seg_pos={0 0}, upload=494438]: Segment reached optimal threshold of 13 pieces. Dec 29 03:25:13.937375103 (563h56m13.808458138s) [noise=true , address=45.91.116.14:4191 , piece_id=WBXLM4TNAT4SJX3UTU7T36A2IEGJVUHW57PNM3PCAO33UGMXRT4Q, seg_pos={0 0}, upload=494438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 29 03:25:13.937381205 (563h56m13.80846425s) [noise=true , address=45.91.116.14:4191 , piece_id=WBXLM4TNAT4SJX3UTU7T36A2IEGJVUHW57PNM3PCAO33UGMXRT4Q, seg_pos={0 0}, upload=494438]: Long tail cancellation due to optimal threshold Dec 29 03:25:13.937552962 (563h56m13.808636007s) [noise=true , address=77.90.31.13:4627 , piece_id=2XWPNVP7JHGKBYD32EGUG5BECQH6YAXLLMVEKHQB62XDJFKXIJHQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:58080->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:58080->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58080->77.90.31.13:4627: use of closed network connection Dec 29 03:25:13.937561899 (563h56m13.808644944s) [noise=true , address=77.90.31.13:4627 , piece_id=2XWPNVP7JHGKBYD32EGUG5BECQH6YAXLLMVEKHQB62XDJFKXIJHQ, seg_pos={0 0}, upload=494438]: Long tail cancellation due to optimal threshold Dec 29 03:25:13.937591485 (563h56m13.80867452s) [noise=true , address=77.90.31.12:4589 , piece_id=HWDLPR4SNVR7GLKQYNJXSEFF5A4CNWZNY32HUBCWVU4DLRHJALQQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 29 03:25:13.937598710 (563h56m13.808681745s) [noise=true , address=77.90.31.12:4589 , piece_id=HWDLPR4SNVR7GLKQYNJXSEFF5A4CNWZNY32HUBCWVU4DLRHJALQQ, seg_pos={0 0}, upload=494438]: Long tail cancellation due to optimal threshold Dec 29 03:25:13.937618697 (563h56m13.808701732s) [seg_pos={0 0}, upload=494438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:25:13.937685905 (563h56m13.80876894s) [upload=494438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:25:13.937690594 (563h56m13.808773629s) [upload=494438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:25:14.227611579 (563h56m14.098694624s) [upload=494438]: Done uploading segments... Dec 29 03:25:14.227631487 (563h56m14.098714523s) [upload=494438]: Upload finished. err: <nil> Duration: 5.925535449s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [XX..X.X.X....X.X.XX..XX.....XX.X.....X.X..XXXX..XXX.XX...X.X] duration: 30.003879672s trace: https://jaeger.dev.storj.io/trace/fd304c9583ba31a FAIL (2025-12-29 03:25:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 03:25:10.777514077 (563h56m10.648597122s) [upload=494438]: Starting upload Dec 29 03:25:10.777595341 (563h56m10.648678386s) [upload=494438]: Done starting upload Dec 29 03:25:10.777947953 (563h56m10.649030999s) [upload=494438]: Uploading segments... Dec 29 03:25:10.777993450 (563h56m10.649076495s) [upload=494438]: Got next segment. Inline: false Dec 29 03:25:10.778044597 (563h56m10.649127642s) [upload=494438]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:25:10.778049566 (563h56m10.649132601s) [upload=494438]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:25:11.023293341 (563h56m10.894376377s) [seg_pos={0 0}, upload=494438]: Begin upload segment... Dec 29 03:25:11.023338908 (563h56m10.894421953s) [seg_pos={0 0}, upload=494438]: Done begin upload segment. Dec 29 03:25:11.023377561 (563h56m10.894460596s) [noise=true , address=77.90.31.15:4697 , piece_id=6NRWSCWKICUNGEPQ5NKKBJF3AABDMQDIASETPSNXWMBJ2U7GL64A, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.023435903 (563h56m10.894518938s) [noise=true , address=145.79.227.37:4567 , piece_id=6Y7UJKYJZEX3OC2TM37RRWBXAN4QANNWS5HL74VJJQQNC7UVB7SA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.024545235 (563h56m10.89562827s) [noise=true , address=145.79.227.41:4087 , piece_id=T6JZVKG3D45TBA2CI24Q3ZOPKNBZYRQNBUA3THA3PZWSIOX6H3QA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.024761807 (563h56m10.895844843s) [noise=true , address=145.79.227.39:4013 , piece_id=OTXRXVS666KLF74RYWBS6KLW3TUX7RJQLQDMIXIRN43WAS6QOMMQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.025191245 (563h56m10.89627428s) [noise=true , address=145.79.227.50:4375 , piece_id=VFZNMYHCBTCK7DOVEKR6RDGD3KO6K3KJ3RJHGQ4LUM3NHFBLVWYA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.025365376 (563h56m10.896448412s) [noise=true , address=77.90.31.12:4589 , piece_id=HWDLPR4SNVR7GLKQYNJXSEFF5A4CNWZNY32HUBCWVU4DLRHJALQQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.026271602 (563h56m10.897354637s) [noise=true , address=145.79.227.36:4517 , piece_id=KKBCMNEBUQJMOCKPSUQWA5KMCEM4RMLVIXNHLS2WNG22UEQ3TWWQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.026422379 (563h56m10.897505414s) [noise=true , address=145.79.227.19:4079 , piece_id=BNYQNAWKFS3CWRGFTU2WZMD7B3JWBXE6TE73CCOG3UMXCTYFVYDQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.027464674 (563h56m10.898547709s) [noise=true , address=145.79.227.11:4731 , piece_id=YR5EXBN5CMLGR65ABOQJCRPG2UTYNLJZSKPADO3DA45PUJYRAWSA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.027586847 (563h56m10.898669882s) [noise=true , address=45.91.116.14:4191 , piece_id=WBXLM4TNAT4SJX3UTU7T36A2IEGJVUHW57PNM3PCAO33UGMXRT4Q, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.028313140 (563h56m10.899396175s) [noise=true , address=145.79.227.48:4303 , piece_id=2UNUYZ3HHI4HLZXOVSCQXWCUAKPDXE5YLUVBCYEL4IVSMGE4WLVA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.028528690 (563h56m10.899611735s) [noise=true , address=145.79.227.51:4409 , piece_id=AUI6FSNO6YEH5JMSTELCTYBAQCSJCYW36CG56IKDYGT2MYJFFTOQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.028869249 (563h56m10.899952284s) [noise=true , address=45.91.116.5:4299 , piece_id=6KQ23R6NG74TJ472ENMR5C35AWNA5TAW53L3KFB4CZWER3KYTAMQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.029374130 (563h56m10.900457175s) [noise=true , address=77.90.31.11:4555 , piece_id=X43PH3ICCMDXVP73DUVGDLMRQJ3YPF5D4TQMRN5XPCDP55IMFZBQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.029760004 (563h56m10.900843049s) [noise=true , address=77.90.31.13:4627 , piece_id=2XWPNVP7JHGKBYD32EGUG5BECQH6YAXLLMVEKHQB62XDJFKXIJHQ, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:11.029960446 (563h56m10.901043491s) [noise=true , address=45.91.116.10:4084 , piece_id=JCIRURL7UZWCP7KQE3M57FXBDACKOGAKIYMT3LZH2C7WVCLZ7XNA, seg_pos={0 0}, upload=494438]: Uploading piece... Dec 29 03:25:12.720599483 (563h56m12.591682528s) [upload=494438]: Next returned nil segment Dec 29 03:25:12.720617647 (563h56m12.591700693s) [upload=494438]: Waiting for error group managing segments... Dec 29 03:25:13.004770289 (563h56m12.875853334s) [noise=true , address=145.79.227.37:4567 , piece_id=6Y7UJKYJZEX3OC2TM37RRWBXAN4QANNWS5HL74VJJQQNC7UVB7SA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.015495160 (563h56m12.886578205s) [noise=true , address=145.79.227.36:4517 , piece_id=KKBCMNEBUQJMOCKPSUQWA5KMCEM4RMLVIXNHLS2WNG22UEQ3TWWQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.016524962 (563h56m12.887608007s) [noise=true , address=145.79.227.39:4013 , piece_id=OTXRXVS666KLF74RYWBS6KLW3TUX7RJQLQDMIXIRN43WAS6QOMMQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.018057500 (563h56m12.889140545s) [noise=true , address=145.79.227.50:4375 , piece_id=VFZNMYHCBTCK7DOVEKR6RDGD3KO6K3KJ3RJHGQ4LUM3NHFBLVWYA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.019998305 (563h56m12.89108134s) [noise=true , address=145.79.227.41:4087 , piece_id=T6JZVKG3D45TBA2CI24Q3ZOPKNBZYRQNBUA3THA3PZWSIOX6H3QA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.027899229 (563h56m12.898982275s) [noise=true , address=145.79.227.51:4409 , piece_id=AUI6FSNO6YEH5JMSTELCTYBAQCSJCYW36CG56IKDYGT2MYJFFTOQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.030448854 (563h56m12.901531899s) [noise=true , address=145.79.227.11:4731 , piece_id=YR5EXBN5CMLGR65ABOQJCRPG2UTYNLJZSKPADO3DA45PUJYRAWSA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.039699990 (563h56m12.910783025s) [noise=true , address=145.79.227.48:4303 , piece_id=2UNUYZ3HHI4HLZXOVSCQXWCUAKPDXE5YLUVBCYEL4IVSMGE4WLVA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.043482302 (563h56m12.914565337s) [noise=true , address=145.79.227.19:4079 , piece_id=BNYQNAWKFS3CWRGFTU2WZMD7B3JWBXE6TE73CCOG3UMXCTYFVYDQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.201215966 (563h56m13.072299001s) [noise=true , address=77.90.31.11:4555 , piece_id=X43PH3ICCMDXVP73DUVGDLMRQJ3YPF5D4TQMRN5XPCDP55IMFZBQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.415979141 (563h56m13.287062186s) [noise=true , address=45.91.116.10:4084 , piece_id=JCIRURL7UZWCP7KQE3M57FXBDACKOGAKIYMT3LZH2C7WVCLZ7XNA, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.423595834 (563h56m13.294678869s) [noise=true , address=45.91.116.5:4299 , piece_id=6KQ23R6NG74TJ472ENMR5C35AWNA5TAW53L3KFB4CZWER3KYTAMQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.937189360 (563h56m13.808272395s) [noise=true , address=77.90.31.15:4697 , piece_id=6NRWSCWKICUNGEPQ5NKKBJF3AABDMQDIASETPSNXWMBJ2U7GL64A, seg_pos={0 0}, upload=494438]: Done uploading piece. err: <nil> Dec 29 03:25:13.937208195 (563h56m13.80829124s) [seg_pos={0 0}, upload=494438]: Segment reached optimal threshold of 13 pieces. Dec 29 03:25:13.937375103 (563h56m13.808458138s) [noise=true , address=45.91.116.14:4191 , piece_id=WBXLM4TNAT4SJX3UTU7T36A2IEGJVUHW57PNM3PCAO33UGMXRT4Q, seg_pos={0 0}, upload=494438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 29 03:25:13.937381205 (563h56m13.80846425s) [noise=true , address=45.91.116.14:4191 , piece_id=WBXLM4TNAT4SJX3UTU7T36A2IEGJVUHW57PNM3PCAO33UGMXRT4Q, seg_pos={0 0}, upload=494438]: Long tail cancellation due to optimal threshold Dec 29 03:25:13.937552962 (563h56m13.808636007s) [noise=true , address=77.90.31.13:4627 , piece_id=2XWPNVP7JHGKBYD32EGUG5BECQH6YAXLLMVEKHQB62XDJFKXIJHQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:58080->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:58080->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58080->77.90.31.13:4627: use of closed network connection Dec 29 03:25:13.937561899 (563h56m13.808644944s) [noise=true , address=77.90.31.13:4627 , piece_id=2XWPNVP7JHGKBYD32EGUG5BECQH6YAXLLMVEKHQB62XDJFKXIJHQ, seg_pos={0 0}, upload=494438]: Long tail cancellation due to optimal threshold Dec 29 03:25:13.937591485 (563h56m13.80867452s) [noise=true , address=77.90.31.12:4589 , piece_id=HWDLPR4SNVR7GLKQYNJXSEFF5A4CNWZNY32HUBCWVU4DLRHJALQQ, seg_pos={0 0}, upload=494438]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: expected piece hash; context canceled; EOF Dec 29 03:25:13.937598710 (563h56m13.808681745s) [noise=true , address=77.90.31.12:4589 , piece_id=HWDLPR4SNVR7GLKQYNJXSEFF5A4CNWZNY32HUBCWVU4DLRHJALQQ, seg_pos={0 0}, upload=494438]: Long tail cancellation due to optimal threshold Dec 29 03:25:13.937618697 (563h56m13.808701732s) [seg_pos={0 0}, upload=494438]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:25:13.937685905 (563h56m13.80876894s) [upload=494438]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:25:13.937690594 (563h56m13.808773629s) [upload=494438]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:25:14.227611579 (563h56m14.098694624s) [upload=494438]: Done uploading segments... Dec 29 03:25:14.227631487 (563h56m14.098714523s) [upload=494438]: Upload finished. err: <nil> Duration: 30.004038806s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....................................................XX......] duration: 3.511131938s trace: https://jaeger.dev.storj.io/trace/481fdf974580574a FAIL (2025-12-29 03:25:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.511162727s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X.................................................X.....] duration: 33.618626858s trace: https://jaeger.dev.storj.io/trace/1584175ba8dba7fe FAIL (2025-12-29 03:24:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ece... Dec 29 03:20:05.592697598 (563h51m5.463780643s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593249339 (563h51m5.464332384s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593687443 (563h51m5.464770488s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594173598 (563h51m5.465256633s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594639957 (563h51m5.465723002s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595331994 (563h51m5.466415039s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595834151 (563h51m5.466917206s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.596329844 (563h51m5.467412899s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599128643 (563h51m5.470211688s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599196953 (563h51m5.470279988s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.601094367 (563h51m5.472177412s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.602868896 (563h51m5.473951941s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.604661328 (563h51m5.475744374s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.606462138 (563h51m5.477545183s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.609580124 (563h51m5.480663169s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.611644807 (563h51m5.482727852s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:06.901416710 (563h51m6.772499755s) [upload=494362]: Next returned nil segment Dec 29 03:20:06.901434123 (563h51m6.772517168s) [upload=494362]: Waiting for error group managing segments... Dec 29 03:20:07.237383793 (563h51m7.108466848s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.439111607 (563h51m7.310194652s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.462301153 (563h51m7.333384198s) [noise=true , address=145.79.227.21:4498 , piece_id=RTK3PTWGWTIQSNCGREQB37OID2RGK37QWEKTDYFQKASGH6SZCURA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.479891718 (563h51m7.350974763s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.486062297 (563h51m7.357145342s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.578667930 (563h51m7.449750975s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.701241555 (563h51m7.5723246s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.745868671 (563h51m7.616951706s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.799936903 (563h51m7.671019948s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.430929511 (563h51m8.302012556s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.432106221 (563h51m8.303189256s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.447633778 (563h51m8.318716813s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.459472520 (563h51m8.330555555s) [noise=true , address=145.79.227.34:4916 , piece_id=4U67LINR5A4GR2NIUVMB2IPTT5DJQOYHM6HDYKZEICKF5JY53VFA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499215165 (563h51m8.37029821s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499245653 (563h51m8.370328698s) [seg_pos={0 0}, upload=494362]: Segment reached optimal threshold of 14 pieces. Dec 29 03:20:08.499544722 (563h51m8.370627797s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.499555243 (563h51m8.370638288s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.499833272 (563h51m8.370916327s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection Dec 29 03:20:08.499843291 (563h51m8.370926366s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500107964 (563h51m8.371191019s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.500166165 (563h51m8.37124921s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500868092 (563h51m8.371951147s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53): protocol: context canceled; context canceled Dec 29 03:20:08.500880466 (563h51m8.371963511s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500945178 (563h51m8.372028223s) [seg_pos={0 0}, upload=494362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:20:08.501128358 (563h51m8.372211413s) [upload=494362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:20:08.501156331 (563h51m8.372239376s) [upload=494362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:20:08.795508553 (563h51m8.666591598s) [upload=494362]: Done uploading segments... Dec 29 03:20:08.795532639 (563h51m8.666615684s) [upload=494362]: Upload finished. err: <nil> Duration: 33.618644271s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....................................................XX.....] duration: 3.516723748s trace: https://jaeger.dev.storj.io/trace/481fdf974580574a FAIL (2025-12-29 03:24:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.516749147s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X.................................................X....] duration: 39.625084587s trace: https://jaeger.dev.storj.io/trace/1584175ba8dba7fe FAIL (2025-12-29 03:24:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ece... Dec 29 03:20:05.592697598 (563h51m5.463780643s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593249339 (563h51m5.464332384s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593687443 (563h51m5.464770488s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594173598 (563h51m5.465256633s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594639957 (563h51m5.465723002s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595331994 (563h51m5.466415039s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595834151 (563h51m5.466917206s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.596329844 (563h51m5.467412899s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599128643 (563h51m5.470211688s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599196953 (563h51m5.470279988s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.601094367 (563h51m5.472177412s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.602868896 (563h51m5.473951941s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.604661328 (563h51m5.475744374s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.606462138 (563h51m5.477545183s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.609580124 (563h51m5.480663169s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.611644807 (563h51m5.482727852s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:06.901416710 (563h51m6.772499755s) [upload=494362]: Next returned nil segment Dec 29 03:20:06.901434123 (563h51m6.772517168s) [upload=494362]: Waiting for error group managing segments... Dec 29 03:20:07.237383793 (563h51m7.108466848s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.439111607 (563h51m7.310194652s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.462301153 (563h51m7.333384198s) [noise=true , address=145.79.227.21:4498 , piece_id=RTK3PTWGWTIQSNCGREQB37OID2RGK37QWEKTDYFQKASGH6SZCURA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.479891718 (563h51m7.350974763s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.486062297 (563h51m7.357145342s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.578667930 (563h51m7.449750975s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.701241555 (563h51m7.5723246s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.745868671 (563h51m7.616951706s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.799936903 (563h51m7.671019948s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.430929511 (563h51m8.302012556s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.432106221 (563h51m8.303189256s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.447633778 (563h51m8.318716813s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.459472520 (563h51m8.330555555s) [noise=true , address=145.79.227.34:4916 , piece_id=4U67LINR5A4GR2NIUVMB2IPTT5DJQOYHM6HDYKZEICKF5JY53VFA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499215165 (563h51m8.37029821s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499245653 (563h51m8.370328698s) [seg_pos={0 0}, upload=494362]: Segment reached optimal threshold of 14 pieces. Dec 29 03:20:08.499544722 (563h51m8.370627797s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.499555243 (563h51m8.370638288s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.499833272 (563h51m8.370916327s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection Dec 29 03:20:08.499843291 (563h51m8.370926366s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500107964 (563h51m8.371191019s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.500166165 (563h51m8.37124921s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500868092 (563h51m8.371951147s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53): protocol: context canceled; context canceled Dec 29 03:20:08.500880466 (563h51m8.371963511s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500945178 (563h51m8.372028223s) [seg_pos={0 0}, upload=494362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:20:08.501128358 (563h51m8.372211413s) [upload=494362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:20:08.501156331 (563h51m8.372239376s) [upload=494362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:20:08.795508553 (563h51m8.666591598s) [upload=494362]: Done uploading segments... Dec 29 03:20:08.795532639 (563h51m8.666615684s) [upload=494362]: Upload finished. err: <nil> Duration: 39.625110766s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.XX..X.X.X....X.X.XX..XX.....XX.X.....X.X..XXXX..XXX.XX...X.] duration: 5.962505205s trace: https://jaeger.dev.storj.io/trace/29f5e30ea54e0e11 FAIL (2025-12-29 03:24:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 03:23:00.423908552 (563h54m0.294991587s) [upload=494406]: Starting upload Dec 29 03:23:00.423986470 (563h54m0.295069525s) [upload=494406]: Done starting upload Dec 29 03:23:00.424142898 (563h54m0.295225953s) [upload=494406]: Uploading segments... Dec 29 03:23:00.424265641 (563h54m0.295348676s) [upload=494406]: Got next segment. Inline: false Dec 29 03:23:00.424306720 (563h54m0.295389765s) [upload=494406]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:23:00.424312561 (563h54m0.295395606s) [upload=494406]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:23:00.666015594 (563h54m0.537098639s) [seg_pos={0 0}, upload=494406]: Begin upload segment... Dec 29 03:23:00.666090968 (563h54m0.537174003s) [seg_pos={0 0}, upload=494406]: Done begin upload segment. Dec 29 03:23:00.666161462 (563h54m0.537244507s) [noise=true , address=45.91.116.7:4012 , piece_id=PGQXBIWAIN2XIRPEHEU7RAF3WCYCXEOHMH2ZVCOIX4HYEICL34SQ, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.666669039 (563h54m0.537752084s) [noise=true , address=145.79.227.36:4518 , piece_id=QK4LSFSCTX3KC363CZJZX5KGRK3M6FGDFY3EEVYE35Z5GZMKE5LQ, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.667179370 (563h54m0.538262405s) [noise=true , address=45.91.116.14:4190 , piece_id=AFQO4ILUMUW3756CW4TEMGIDI4O3E54WAI54STRGBB36ODKIB4BQ, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.667641770 (563h54m0.538724815s) [noise=true , address=145.79.227.52:4445 , piece_id=CFTHUC77CZJCHBVIIXBSFLJN47WDCXTRHHYKHBOFYUGZNJECRFVA, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.668056700 (563h54m0.539139745s) [noise=true , address=45.91.116.10:4081 , piece_id=TNKZQBIOVQ5RABRBELRAVPEMYPD5NJD5K3Z4DAGNVLXHBX7QGV3A, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.668480868 (563h54m0.539563913s) [noise=true , address=77.90.31.21:4880 , piece_id=YTHMSONLUS5D43N6FFZAJVCIZVFUFSAWZAJURMGVE2UENFXLJB6Q, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.668886040 (563h54m0.539969075s) [noise=true , address=145.79.227.46:4601 , piece_id=MH4JXEVU35A6CBNP52AGXALIBPOYZZAQP6E3LNDK25PIUFFHZ3HA, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.669359251 (563h54m0.540442286s) [noise=true , address=145.79.227.19:4077 , piece_id=KKMNC2SFLQDI4YP5VQLZD26CP3T2NGW3FYWFBTUEVX3S24DWR36A, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.670462291 (563h54m0.541545326s) [noise=true , address=145.79.227.15:4876 , piece_id=LH2PLG6GSJL3XSFDO6XYFNE7KY6NTP3WRAKAUBZ4NEWWWPGV2MMQ, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.671732480 (563h54m0.542815515s) [noise=true , address=45.91.116.6:4975 , piece_id=TLNOWZK572WKXLBOFFOESMVRETB5EHVFNK4XILNJJHZEIDFCCRKQ, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.672110159 (563h54m0.543193194s) [noise=true , address=77.90.31.18:4805 , piece_id=LC2KVRNGNN6GTHSLKFGNKPSGDFI3V4PVZJTDX3QKYTCBKQHML3UA, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.673556854 (563h54m0.544639899s) [noise=true , address=45.91.116.9:4264 , piece_id=SPVKMUJ3LKPUR6BDIJIAI5W2PW6U6F2NRFOOR2UDFSQWHM7X5QFQ, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.676760584 (563h54m0.547843619s) [noise=true , address=145.79.227.12:4767 , piece_id=OPZ6ESVBVCYR35T2TKBCGPBRWD2IWLA3ZCFZ24DJJO6KLJRGOOBA, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.678919695 (563h54m0.55000273s) [noise=true , address=145.79.227.45:4231 , piece_id=4HNTLT5D6WBBK4W6V4HFMVEHW7OHQEIM4SJLWDAREK234OCPGHFA, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.680330601 (563h54m0.551413636s) [noise=true , address=145.79.227.2:4115 , piece_id=AH7E6R4R5M6YZ45CHM5F34BCRIJ66TEVDD6DAUW24U6CGZ2RYXDA, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.680833299 (563h54m0.551916334s) [noise=true , address=145.79.227.13:4802 , piece_id=RIJAFV2U3PPARYRD6GTRXE2YTFN3ZBD2NXMYBTVUK5ME5FB4VCPA, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:02.290924598 (563h54m2.162007663s) [upload=494406]: Next returned nil segment Dec 29 03:23:02.290939015 (563h54m2.16202206s) [upload=494406]: Waiting for error group managing segments... Dec 29 03:23:02.670648359 (563h54m2.541731404s) [noise=true , address=145.79.227.15:4876 , piece_id=LH2PLG6GSJL3XSFDO6XYFNE7KY6NTP3WRAKAUBZ4NEWWWPGV2MMQ, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:02.671044663 (563h54m2.542127708s) [noise=true , address=145.79.227.12:4767 , piece_id=OPZ6ESVBVCYR35T2TKBCGPBRWD2IWLA3ZCFZ24DJJO6KLJRGOOBA, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:02.680164559 (563h54m2.551247594s) [noise=true , address=145.79.227.13:4802 , piece_id=RIJAFV2U3PPARYRD6GTRXE2YTFN3ZBD2NXMYBTVUK5ME5FB4VCPA, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:02.682448858 (563h54m2.553531903s) [noise=true , address=145.79.227.52:4445 , piece_id=CFTHUC77CZJCHBVIIXBSFLJN47WDCXTRHHYKHBOFYUGZNJECRFVA, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:02.683282345 (563h54m2.55436539s) [noise=true , address=145.79.227.2:4115 , piece_id=AH7E6R4R5M6YZ45CHM5F34BCRIJ66TEVDD6DAUW24U6CGZ2RYXDA, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:02.686453123 (563h54m2.557536168s) [noise=true , address=145.79.227.36:4518 , piece_id=QK4LSFSCTX3KC363CZJZX5KGRK3M6FGDFY3EEVYE35Z5GZMKE5LQ, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:02.686474604 (563h54m2.557557649s) [noise=true , address=145.79.227.46:4601 , piece_id=MH4JXEVU35A6CBNP52AGXALIBPOYZZAQP6E3LNDK25PIUFFHZ3HA, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:02.696583893 (563h54m2.567666939s) [noise=true , address=145.79.227.45:4231 , piece_id=4HNTLT5D6WBBK4W6V4HFMVEHW7OHQEIM4SJLWDAREK234OCPGHFA, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:02.722828235 (563h54m2.59391128s) [noise=true , address=145.79.227.19:4077 , piece_id=KKMNC2SFLQDI4YP5VQLZD26CP3T2NGW3FYWFBTUEVX3S24DWR36A, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:02.888880961 (563h54m2.759964006s) [noise=true , address=45.91.116.14:4190 , piece_id=AFQO4ILUMUW3756CW4TEMGIDI4O3E54WAI54STRGBB36ODKIB4BQ, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:02.954924927 (563h54m2.826007972s) [noise=true , address=77.90.31.21:4880 , piece_id=YTHMSONLUS5D43N6FFZAJVCIZVFUFSAWZAJURMGVE2UENFXLJB6Q, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:02.969743262 (563h54m2.840826307s) [noise=true , address=45.91.116.7:4012 , piece_id=PGQXBIWAIN2XIRPEHEU7RAF3WCYCXEOHMH2ZVCOIX4HYEICL34SQ, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:03.019432146 (563h54m2.890515191s) [noise=true , address=45.91.116.10:4081 , piece_id=TNKZQBIOVQ5RABRBELRAVPEMYPD5NJD5K3Z4DAGNVLXHBX7QGV3A, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:03.019461662 (563h54m2.890544707s) [seg_pos={0 0}, upload=494406]: Segment reached optimal threshold of 13 pieces. Dec 29 03:23:03.019745803 (563h54m2.890828848s) [noise=true , address=45.91.116.6:4975 , piece_id=TLNOWZK572WKXLBOFFOESMVRETB5EHVFNK4XILNJJHZEIDFCCRKQ, seg_pos={0 0}, upload=494406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 29 03:23:03.019759089 (563h54m2.890842164s) [noise=true , address=45.91.116.6:4975 , piece_id=TLNOWZK572WKXLBOFFOESMVRETB5EHVFNK4XILNJJHZEIDFCCRKQ, seg_pos={0 0}, upload=494406]: Long tail cancellation due to optimal threshold Dec 29 03:23:03.019965993 (563h54m2.891049048s) [noise=true , address=45.91.116.9:4264 , piece_id=SPVKMUJ3LKPUR6BDIJIAI5W2PW6U6F2NRFOOR2UDFSQWHM7X5QFQ, seg_pos={0 0}, upload=494406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 29 03:23:03.019976102 (563h54m2.891059147s) [noise=true , address=45.91.116.9:4264 , piece_id=SPVKMUJ3LKPUR6BDIJIAI5W2PW6U6F2NRFOOR2UDFSQWHM7X5QFQ, seg_pos={0 0}, upload=494406]: Long tail cancellation due to optimal threshold Dec 29 03:23:03.020081803 (563h54m2.891164858s) [noise=true , address=77.90.31.18:4805 , piece_id=LC2KVRNGNN6GTHSLKFGNKPSGDFI3V4PVZJTDX3QKYTCBKQHML3UA, seg_pos={0 0}, upload=494406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 29 03:23:03.020103645 (563h54m2.89118669s) [noise=true , address=77.90.31.18:4805 , piece_id=LC2KVRNGNN6GTHSLKFGNKPSGDFI3V4PVZJTDX3QKYTCBKQHML3UA, seg_pos={0 0}, upload=494406]: Long tail cancellation due to optimal threshold Dec 29 03:23:03.020135355 (563h54m2.89121839s) [seg_pos={0 0}, upload=494406]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:23:03.020259231 (563h54m2.891342276s) [upload=494406]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:23:03.020266415 (563h54m2.89134945s) [upload=494406]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:23:03.304811753 (563h54m3.175894828s) [upload=494406]: Done uploading segments... Dec 29 03:23:03.304836951 (563h54m3.175920016s) [upload=494406]: Upload finished. err: <nil> Duration: 5.962531574s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......................................................XX....] duration: 5.001680382s trace: https://jaeger.dev.storj.io/trace/481fdf974580574a FAIL (2025-12-29 03:23:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001697523s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [..XX..X.X.X....X.X.XX..XX.....XX.X.....X.X..XXXX..XXX.XX...X] duration: 30.003016491s trace: https://jaeger.dev.storj.io/trace/29f5e30ea54e0e11 FAIL (2025-12-29 03:23:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 29 03:23:00.423908552 (563h54m0.294991587s) [upload=494406]: Starting upload Dec 29 03:23:00.423986470 (563h54m0.295069525s) [upload=494406]: Done starting upload Dec 29 03:23:00.424142898 (563h54m0.295225953s) [upload=494406]: Uploading segments... Dec 29 03:23:00.424265641 (563h54m0.295348676s) [upload=494406]: Got next segment. Inline: false Dec 29 03:23:00.424306720 (563h54m0.295389765s) [upload=494406]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:23:00.424312561 (563h54m0.295395606s) [upload=494406]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:23:00.666015594 (563h54m0.537098639s) [seg_pos={0 0}, upload=494406]: Begin upload segment... Dec 29 03:23:00.666090968 (563h54m0.537174003s) [seg_pos={0 0}, upload=494406]: Done begin upload segment. Dec 29 03:23:00.666161462 (563h54m0.537244507s) [noise=true , address=45.91.116.7:4012 , piece_id=PGQXBIWAIN2XIRPEHEU7RAF3WCYCXEOHMH2ZVCOIX4HYEICL34SQ, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.666669039 (563h54m0.537752084s) [noise=true , address=145.79.227.36:4518 , piece_id=QK4LSFSCTX3KC363CZJZX5KGRK3M6FGDFY3EEVYE35Z5GZMKE5LQ, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.667179370 (563h54m0.538262405s) [noise=true , address=45.91.116.14:4190 , piece_id=AFQO4ILUMUW3756CW4TEMGIDI4O3E54WAI54STRGBB36ODKIB4BQ, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.667641770 (563h54m0.538724815s) [noise=true , address=145.79.227.52:4445 , piece_id=CFTHUC77CZJCHBVIIXBSFLJN47WDCXTRHHYKHBOFYUGZNJECRFVA, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.668056700 (563h54m0.539139745s) [noise=true , address=45.91.116.10:4081 , piece_id=TNKZQBIOVQ5RABRBELRAVPEMYPD5NJD5K3Z4DAGNVLXHBX7QGV3A, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.668480868 (563h54m0.539563913s) [noise=true , address=77.90.31.21:4880 , piece_id=YTHMSONLUS5D43N6FFZAJVCIZVFUFSAWZAJURMGVE2UENFXLJB6Q, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.668886040 (563h54m0.539969075s) [noise=true , address=145.79.227.46:4601 , piece_id=MH4JXEVU35A6CBNP52AGXALIBPOYZZAQP6E3LNDK25PIUFFHZ3HA, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.669359251 (563h54m0.540442286s) [noise=true , address=145.79.227.19:4077 , piece_id=KKMNC2SFLQDI4YP5VQLZD26CP3T2NGW3FYWFBTUEVX3S24DWR36A, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.670462291 (563h54m0.541545326s) [noise=true , address=145.79.227.15:4876 , piece_id=LH2PLG6GSJL3XSFDO6XYFNE7KY6NTP3WRAKAUBZ4NEWWWPGV2MMQ, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.671732480 (563h54m0.542815515s) [noise=true , address=45.91.116.6:4975 , piece_id=TLNOWZK572WKXLBOFFOESMVRETB5EHVFNK4XILNJJHZEIDFCCRKQ, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.672110159 (563h54m0.543193194s) [noise=true , address=77.90.31.18:4805 , piece_id=LC2KVRNGNN6GTHSLKFGNKPSGDFI3V4PVZJTDX3QKYTCBKQHML3UA, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.673556854 (563h54m0.544639899s) [noise=true , address=45.91.116.9:4264 , piece_id=SPVKMUJ3LKPUR6BDIJIAI5W2PW6U6F2NRFOOR2UDFSQWHM7X5QFQ, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.676760584 (563h54m0.547843619s) [noise=true , address=145.79.227.12:4767 , piece_id=OPZ6ESVBVCYR35T2TKBCGPBRWD2IWLA3ZCFZ24DJJO6KLJRGOOBA, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.678919695 (563h54m0.55000273s) [noise=true , address=145.79.227.45:4231 , piece_id=4HNTLT5D6WBBK4W6V4HFMVEHW7OHQEIM4SJLWDAREK234OCPGHFA, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.680330601 (563h54m0.551413636s) [noise=true , address=145.79.227.2:4115 , piece_id=AH7E6R4R5M6YZ45CHM5F34BCRIJ66TEVDD6DAUW24U6CGZ2RYXDA, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:00.680833299 (563h54m0.551916334s) [noise=true , address=145.79.227.13:4802 , piece_id=RIJAFV2U3PPARYRD6GTRXE2YTFN3ZBD2NXMYBTVUK5ME5FB4VCPA, seg_pos={0 0}, upload=494406]: Uploading piece... Dec 29 03:23:02.290924598 (563h54m2.162007663s) [upload=494406]: Next returned nil segment Dec 29 03:23:02.290939015 (563h54m2.16202206s) [upload=494406]: Waiting for error group managing segments... Dec 29 03:23:02.670648359 (563h54m2.541731404s) [noise=true , address=145.79.227.15:4876 , piece_id=LH2PLG6GSJL3XSFDO6XYFNE7KY6NTP3WRAKAUBZ4NEWWWPGV2MMQ, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:02.671044663 (563h54m2.542127708s) [noise=true , address=145.79.227.12:4767 , piece_id=OPZ6ESVBVCYR35T2TKBCGPBRWD2IWLA3ZCFZ24DJJO6KLJRGOOBA, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:02.680164559 (563h54m2.551247594s) [noise=true , address=145.79.227.13:4802 , piece_id=RIJAFV2U3PPARYRD6GTRXE2YTFN3ZBD2NXMYBTVUK5ME5FB4VCPA, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:02.682448858 (563h54m2.553531903s) [noise=true , address=145.79.227.52:4445 , piece_id=CFTHUC77CZJCHBVIIXBSFLJN47WDCXTRHHYKHBOFYUGZNJECRFVA, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:02.683282345 (563h54m2.55436539s) [noise=true , address=145.79.227.2:4115 , piece_id=AH7E6R4R5M6YZ45CHM5F34BCRIJ66TEVDD6DAUW24U6CGZ2RYXDA, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:02.686453123 (563h54m2.557536168s) [noise=true , address=145.79.227.36:4518 , piece_id=QK4LSFSCTX3KC363CZJZX5KGRK3M6FGDFY3EEVYE35Z5GZMKE5LQ, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:02.686474604 (563h54m2.557557649s) [noise=true , address=145.79.227.46:4601 , piece_id=MH4JXEVU35A6CBNP52AGXALIBPOYZZAQP6E3LNDK25PIUFFHZ3HA, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:02.696583893 (563h54m2.567666939s) [noise=true , address=145.79.227.45:4231 , piece_id=4HNTLT5D6WBBK4W6V4HFMVEHW7OHQEIM4SJLWDAREK234OCPGHFA, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:02.722828235 (563h54m2.59391128s) [noise=true , address=145.79.227.19:4077 , piece_id=KKMNC2SFLQDI4YP5VQLZD26CP3T2NGW3FYWFBTUEVX3S24DWR36A, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:02.888880961 (563h54m2.759964006s) [noise=true , address=45.91.116.14:4190 , piece_id=AFQO4ILUMUW3756CW4TEMGIDI4O3E54WAI54STRGBB36ODKIB4BQ, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:02.954924927 (563h54m2.826007972s) [noise=true , address=77.90.31.21:4880 , piece_id=YTHMSONLUS5D43N6FFZAJVCIZVFUFSAWZAJURMGVE2UENFXLJB6Q, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:02.969743262 (563h54m2.840826307s) [noise=true , address=45.91.116.7:4012 , piece_id=PGQXBIWAIN2XIRPEHEU7RAF3WCYCXEOHMH2ZVCOIX4HYEICL34SQ, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:03.019432146 (563h54m2.890515191s) [noise=true , address=45.91.116.10:4081 , piece_id=TNKZQBIOVQ5RABRBELRAVPEMYPD5NJD5K3Z4DAGNVLXHBX7QGV3A, seg_pos={0 0}, upload=494406]: Done uploading piece. err: <nil> Dec 29 03:23:03.019461662 (563h54m2.890544707s) [seg_pos={0 0}, upload=494406]: Segment reached optimal threshold of 13 pieces. Dec 29 03:23:03.019745803 (563h54m2.890828848s) [noise=true , address=45.91.116.6:4975 , piece_id=TLNOWZK572WKXLBOFFOESMVRETB5EHVFNK4XILNJJHZEIDFCCRKQ, seg_pos={0 0}, upload=494406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 29 03:23:03.019759089 (563h54m2.890842164s) [noise=true , address=45.91.116.6:4975 , piece_id=TLNOWZK572WKXLBOFFOESMVRETB5EHVFNK4XILNJJHZEIDFCCRKQ, seg_pos={0 0}, upload=494406]: Long tail cancellation due to optimal threshold Dec 29 03:23:03.019965993 (563h54m2.891049048s) [noise=true , address=45.91.116.9:4264 , piece_id=SPVKMUJ3LKPUR6BDIJIAI5W2PW6U6F2NRFOOR2UDFSQWHM7X5QFQ, seg_pos={0 0}, upload=494406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 29 03:23:03.019976102 (563h54m2.891059147s) [noise=true , address=45.91.116.9:4264 , piece_id=SPVKMUJ3LKPUR6BDIJIAI5W2PW6U6F2NRFOOR2UDFSQWHM7X5QFQ, seg_pos={0 0}, upload=494406]: Long tail cancellation due to optimal threshold Dec 29 03:23:03.020081803 (563h54m2.891164858s) [noise=true , address=77.90.31.18:4805 , piece_id=LC2KVRNGNN6GTHSLKFGNKPSGDFI3V4PVZJTDX3QKYTCBKQHML3UA, seg_pos={0 0}, upload=494406]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 29 03:23:03.020103645 (563h54m2.89118669s) [noise=true , address=77.90.31.18:4805 , piece_id=LC2KVRNGNN6GTHSLKFGNKPSGDFI3V4PVZJTDX3QKYTCBKQHML3UA, seg_pos={0 0}, upload=494406]: Long tail cancellation due to optimal threshold Dec 29 03:23:03.020135355 (563h54m2.89121839s) [seg_pos={0 0}, upload=494406]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:23:03.020259231 (563h54m2.891342276s) [upload=494406]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:23:03.020266415 (563h54m2.89134945s) [upload=494406]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:23:03.304811753 (563h54m3.175894828s) [upload=494406]: Done uploading segments... Dec 29 03:23:03.304836951 (563h54m3.175920016s) [upload=494406]: Upload finished. err: <nil> Duration: 30.003036819s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X.................................................X...] duration: 33.443997154s trace: https://jaeger.dev.storj.io/trace/1584175ba8dba7fe FAIL (2025-12-29 03:23:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ece... Dec 29 03:20:05.592697598 (563h51m5.463780643s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593249339 (563h51m5.464332384s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593687443 (563h51m5.464770488s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594173598 (563h51m5.465256633s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594639957 (563h51m5.465723002s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595331994 (563h51m5.466415039s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595834151 (563h51m5.466917206s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.596329844 (563h51m5.467412899s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599128643 (563h51m5.470211688s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599196953 (563h51m5.470279988s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.601094367 (563h51m5.472177412s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.602868896 (563h51m5.473951941s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.604661328 (563h51m5.475744374s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.606462138 (563h51m5.477545183s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.609580124 (563h51m5.480663169s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.611644807 (563h51m5.482727852s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:06.901416710 (563h51m6.772499755s) [upload=494362]: Next returned nil segment Dec 29 03:20:06.901434123 (563h51m6.772517168s) [upload=494362]: Waiting for error group managing segments... Dec 29 03:20:07.237383793 (563h51m7.108466848s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.439111607 (563h51m7.310194652s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.462301153 (563h51m7.333384198s) [noise=true , address=145.79.227.21:4498 , piece_id=RTK3PTWGWTIQSNCGREQB37OID2RGK37QWEKTDYFQKASGH6SZCURA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.479891718 (563h51m7.350974763s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.486062297 (563h51m7.357145342s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.578667930 (563h51m7.449750975s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.701241555 (563h51m7.5723246s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.745868671 (563h51m7.616951706s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.799936903 (563h51m7.671019948s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.430929511 (563h51m8.302012556s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.432106221 (563h51m8.303189256s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.447633778 (563h51m8.318716813s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.459472520 (563h51m8.330555555s) [noise=true , address=145.79.227.34:4916 , piece_id=4U67LINR5A4GR2NIUVMB2IPTT5DJQOYHM6HDYKZEICKF5JY53VFA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499215165 (563h51m8.37029821s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499245653 (563h51m8.370328698s) [seg_pos={0 0}, upload=494362]: Segment reached optimal threshold of 14 pieces. Dec 29 03:20:08.499544722 (563h51m8.370627797s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.499555243 (563h51m8.370638288s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.499833272 (563h51m8.370916327s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection Dec 29 03:20:08.499843291 (563h51m8.370926366s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500107964 (563h51m8.371191019s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.500166165 (563h51m8.37124921s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500868092 (563h51m8.371951147s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53): protocol: context canceled; context canceled Dec 29 03:20:08.500880466 (563h51m8.371963511s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500945178 (563h51m8.372028223s) [seg_pos={0 0}, upload=494362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:20:08.501128358 (563h51m8.372211413s) [upload=494362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:20:08.501156331 (563h51m8.372239376s) [upload=494362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:20:08.795508553 (563h51m8.666591598s) [upload=494362]: Done uploading segments... Dec 29 03:20:08.795532639 (563h51m8.666615684s) [upload=494362]: Upload finished. err: <nil> Duration: 33.444013905s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......................................................XX...] duration: 5.000886117s trace: https://jaeger.dev.storj.io/trace/481fdf974580574a FAIL (2025-12-29 03:22:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000901966s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X.................................................X..] duration: 40.067359977s trace: https://jaeger.dev.storj.io/trace/1584175ba8dba7fe FAIL (2025-12-29 03:22:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ece... Dec 29 03:20:05.592697598 (563h51m5.463780643s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593249339 (563h51m5.464332384s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593687443 (563h51m5.464770488s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594173598 (563h51m5.465256633s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594639957 (563h51m5.465723002s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595331994 (563h51m5.466415039s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595834151 (563h51m5.466917206s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.596329844 (563h51m5.467412899s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599128643 (563h51m5.470211688s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599196953 (563h51m5.470279988s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.601094367 (563h51m5.472177412s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.602868896 (563h51m5.473951941s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.604661328 (563h51m5.475744374s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.606462138 (563h51m5.477545183s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.609580124 (563h51m5.480663169s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.611644807 (563h51m5.482727852s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:06.901416710 (563h51m6.772499755s) [upload=494362]: Next returned nil segment Dec 29 03:20:06.901434123 (563h51m6.772517168s) [upload=494362]: Waiting for error group managing segments... Dec 29 03:20:07.237383793 (563h51m7.108466848s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.439111607 (563h51m7.310194652s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.462301153 (563h51m7.333384198s) [noise=true , address=145.79.227.21:4498 , piece_id=RTK3PTWGWTIQSNCGREQB37OID2RGK37QWEKTDYFQKASGH6SZCURA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.479891718 (563h51m7.350974763s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.486062297 (563h51m7.357145342s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.578667930 (563h51m7.449750975s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.701241555 (563h51m7.5723246s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.745868671 (563h51m7.616951706s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.799936903 (563h51m7.671019948s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.430929511 (563h51m8.302012556s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.432106221 (563h51m8.303189256s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.447633778 (563h51m8.318716813s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.459472520 (563h51m8.330555555s) [noise=true , address=145.79.227.34:4916 , piece_id=4U67LINR5A4GR2NIUVMB2IPTT5DJQOYHM6HDYKZEICKF5JY53VFA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499215165 (563h51m8.37029821s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499245653 (563h51m8.370328698s) [seg_pos={0 0}, upload=494362]: Segment reached optimal threshold of 14 pieces. Dec 29 03:20:08.499544722 (563h51m8.370627797s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.499555243 (563h51m8.370638288s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.499833272 (563h51m8.370916327s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection Dec 29 03:20:08.499843291 (563h51m8.370926366s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500107964 (563h51m8.371191019s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.500166165 (563h51m8.37124921s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500868092 (563h51m8.371951147s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53): protocol: context canceled; context canceled Dec 29 03:20:08.500880466 (563h51m8.371963511s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500945178 (563h51m8.372028223s) [seg_pos={0 0}, upload=494362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:20:08.501128358 (563h51m8.372211413s) [upload=494362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:20:08.501156331 (563h51m8.372239376s) [upload=494362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:20:08.795508553 (563h51m8.666591598s) [upload=494362]: Done uploading segments... Dec 29 03:20:08.795532639 (563h51m8.666615684s) [upload=494362]: Upload finished. err: <nil> Duration: 40.067453324s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........................................................XX..] duration: 3.927847796s trace: https://jaeger.dev.storj.io/trace/481fdf974580574a FAIL (2025-12-29 03:22:29 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.927861684s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [X..XX..X.X.X....X.X.XX..XX.....XX.X.....X.X..XXXX..XXX.XX...] duration: 5.900911674s trace: https://jaeger.dev.storj.io/trace/6995403b007d9527 FAIL (2025-12-29 03:22:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: D4GQ2INR5NU7CWIDXRGQXJSU5VIQCW5TEVOCLZHIPPJRT5N6IIQQ) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: NUPN6QWVHXWQEIAS6S2FJZZ2O7MTMODZBGIDJE5IJH4C6SLQQE5Q) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: GENODWOJOMCNHGVLBENXTAEM5SSIQTJLCJ5KJY7WNYW4V7CJARRA) context deadline exceeded ======LOGS====== Dec 29 03:19:34.142993454 (563h50m34.01407649s) [upload=494352]: Starting upload Dec 29 03:19:34.143062005 (563h50m34.01414505s) [upload=494352]: Done starting upload Dec 29 03:19:34.143139853 (563h50m34.014222888s) [upload=494352]: Uploading segments... Dec 29 03:19:34.143244613 (563h50m34.014327648s) [upload=494352]: Got next segment. Inline: false Dec 29 03:19:34.143272265 (563h50m34.0143553s) [upload=494352]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:19:34.143276754 (563h50m34.014359789s) [upload=494352]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:19:34.387054085 (563h50m34.25813713s) [seg_pos={0 0}, upload=494352]: Begin upload segment... Dec 29 03:19:34.387109671 (563h50m34.258192716s) [seg_pos={0 0}, upload=494352]: Done begin upload segment. Dec 29 03:19:34.387143886 (563h50m34.258226921s) [noise=true , address=145.79.227.42:4122 , piece_id=SIWOCNVJLLJXYBGNAHX36SHGDEBJE6HQKMQIGBLRQ4BAGWLCU75A, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.387305193 (563h50m34.258388228s) [noise=true , address=145.79.227.51:4411 , piece_id=NVEARSYSZ2WX2CVMC2MUD6QNLWGIQE3OU74E6OOM654X3OHQWJ6Q, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.388094737 (563h50m34.259177812s) [noise=true , address=145.79.227.35:4481 , piece_id=Q5SC5ODDEY5WIG5PPXZLJDDUAKGMURXU3VNNDUBR4A5ID4JJNZHA, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.388604898 (563h50m34.259687944s) [noise=true , address=45.91.116.8:4045 , piece_id=R6YX47QZWG5XANY4WCPDIJC4UBX4WZT2YINYHA4HDIG4YQER5OGQ, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.389037642 (563h50m34.260120687s) [noise=true , address=145.79.227.38:4978 , piece_id=J7RLKPC73H65EQBHJV6NQZ3RUQSFR4NF5HUH6LRQEMWBGKEUT47A, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.389458624 (563h50m34.260541669s) [noise=true , address=77.90.31.14:4663 , piece_id=D4GQ2INR5NU7CWIDXRGQXJSU5VIQCW5TEVOCLZHIPPJRT5N6IIQQ, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.389731343 (563h50m34.260814378s) [noise=true , address=77.90.31.19:4842 , piece_id=NG4YNMOQVZCPOPSXBTTNFCNP5T6CIQV2XM36IEIFOXAGCTCOWW2Q, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.390038568 (563h50m34.261121603s) [noise=true , address=77.90.31.11:4555 , piece_id=NUPN6QWVHXWQEIAS6S2FJZZ2O7MTMODZBGIDJE5IJH4C6SLQQE5Q, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.390283254 (563h50m34.261366289s) [noise=true , address=145.79.227.12:4765 , piece_id=GHZ4KSLRHU5YBSP4AJHEHWHPUSJ6RW7RCXFE7IEJWN7XTKRMVLIA, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.390322789 (563h50m34.261405824s) [noise=true , address=145.79.227.46:4602 , piece_id=3AZIKLUUQX54BXHAAI3E5WPAGN3D3M7PSYTMHLAWQ7FUNDQ4X6FQ, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.391090130 (563h50m34.262173165s) [noise=true , address=77.90.31.13:4629 , piece_id=GENODWOJOMCNHGVLBENXTAEM5SSIQTJLCJ5KJY7WNYW4V7CJARRA, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.391634356 (563h50m34.262717391s) [noise=true , address=145.79.227.26:4654 , piece_id=OQDDMZC3KYSFULPJ3KIGFA5LLRQVGT3RACMVSOL26RA3V6KXMJRA, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.392876762 (563h50m34.263959807s) [noise=true , address=145.79.227.48:4303 , piece_id=QEHTV45MB2WYGPFNKDA5PV5VE7YMOJTPXT2BEYJU37BLNIOKCBGQ, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.393487676 (563h50m34.264570711s) [noise=true , address=145.79.227.41:4085 , piece_id=BBVGVUIAMGEAI3GFJFFCSJARSFLA66BDU5B6FPOAJ4NWSDL57O6Q, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.393869944 (563h50m34.264952979s) [noise=true , address=145.79.227.50:4375 , piece_id=GES7FXSCHCU4Q6RUJD5AASHHBAVMILVZPCSS4OVITLGNV3HTKVWQ, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.394198078 (563h50m34.265281113s) [noise=true , address=145.79.227.21:4473 , piece_id=TUNQXCRHGCQODQRDHSO7DII2HS4ANYIV3LXUEL7HOPCKNZ3XQ64Q, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:36.076158618 (563h50m35.947241653s) [upload=494352]: Next returned nil segment Dec 29 03:19:36.076175882 (563h50m35.947280597s) [upload=494352]: Waiting for error group managing segments... Dec 29 03:19:36.370782238 (563h50m36.241865283s) [noise=true , address=145.79.227.46:4602 , piece_id=3AZIKLUUQX54BXHAAI3E5WPAGN3D3M7PSYTMHLAWQ7FUNDQ4X6FQ, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.372543873 (563h50m36.243626908s) [noise=true , address=145.79.227.50:4375 , piece_id=GES7FXSCHCU4Q6RUJD5AASHHBAVMILVZPCSS4OVITLGNV3HTKVWQ, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.373146620 (563h50m36.244229655s) [noise=true , address=145.79.227.42:4122 , piece_id=SIWOCNVJLLJXYBGNAHX36SHGDEBJE6HQKMQIGBLRQ4BAGWLCU75A, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.379586383 (563h50m36.250669428s) [noise=true , address=145.79.227.38:4978 , piece_id=J7RLKPC73H65EQBHJV6NQZ3RUQSFR4NF5HUH6LRQEMWBGKEUT47A, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.395642285 (563h50m36.26672532s) [noise=true , address=145.79.227.12:4765 , piece_id=GHZ4KSLRHU5YBSP4AJHEHWHPUSJ6RW7RCXFE7IEJWN7XTKRMVLIA, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.397372199 (563h50m36.268455244s) [noise=true , address=145.79.227.48:4303 , piece_id=QEHTV45MB2WYGPFNKDA5PV5VE7YMOJTPXT2BEYJU37BLNIOKCBGQ, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.399787488 (563h50m36.270870533s) [noise=true , address=145.79.227.21:4473 , piece_id=TUNQXCRHGCQODQRDHSO7DII2HS4ANYIV3LXUEL7HOPCKNZ3XQ64Q, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.403227939 (563h50m36.274310974s) [noise=true , address=145.79.227.41:4085 , piece_id=BBVGVUIAMGEAI3GFJFFCSJARSFLA66BDU5B6FPOAJ4NWSDL57O6Q, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.411425488 (563h50m36.282508533s) [noise=true , address=145.79.227.51:4411 , piece_id=NVEARSYSZ2WX2CVMC2MUD6QNLWGIQE3OU74E6OOM654X3OHQWJ6Q, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.430446266 (563h50m36.301529311s) [noise=true , address=145.79.227.26:4654 , piece_id=OQDDMZC3KYSFULPJ3KIGFA5LLRQVGT3RACMVSOL26RA3V6KXMJRA, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.640650540 (563h50m36.511733575s) [noise=true , address=77.90.31.11:4555 , piece_id=NUPN6QWVHXWQEIAS6S2FJZZ2O7MTMODZBGIDJE5IJH4C6SLQQE5Q, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.663401080 (563h50m36.534484115s) [noise=true , address=77.90.31.13:4629 , piece_id=GENODWOJOMCNHGVLBENXTAEM5SSIQTJLCJ5KJY7WNYW4V7CJARRA, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.678007813 (563h50m36.549090848s) [noise=true , address=77.90.31.14:4663 , piece_id=D4GQ2INR5NU7CWIDXRGQXJSU5VIQCW5TEVOCLZHIPPJRT5N6IIQQ, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.678026729 (563h50m36.549109764s) [seg_pos={0 0}, upload=494352]: Segment reached optimal threshold of 13 pieces. Dec 29 03:19:36.678220327 (563h50m36.549303363s) [noise=true , address=145.79.227.35:4481 , piece_id=Q5SC5ODDEY5WIG5PPXZLJDDUAKGMURXU3VNNDUBR4A5ID4JJNZHA, seg_pos={0 0}, upload=494352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 29 03:19:36.678228783 (563h50m36.549311818s) [noise=true , address=145.79.227.35:4481 , piece_id=Q5SC5ODDEY5WIG5PPXZLJDDUAKGMURXU3VNNDUBR4A5ID4JJNZHA, seg_pos={0 0}, upload=494352]: Long tail cancellation due to optimal threshold Dec 29 03:19:36.678335476 (563h50m36.549418522s) [noise=true , address=77.90.31.19:4842 , piece_id=NG4YNMOQVZCPOPSXBTTNFCNP5T6CIQV2XM36IEIFOXAGCTCOWW2Q, seg_pos={0 0}, upload=494352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 29 03:19:36.678340797 (563h50m36.549423842s) [noise=true , address=77.90.31.19:4842 , piece_id=NG4YNMOQVZCPOPSXBTTNFCNP5T6CIQV2XM36IEIFOXAGCTCOWW2Q, seg_pos={0 0}, upload=494352]: Long tail cancellation due to optimal threshold Dec 29 03:19:36.678576005 (563h50m36.54965904s) [noise=true , address=45.91.116.8:4045 , piece_id=R6YX47QZWG5XANY4WCPDIJC4UBX4WZT2YINYHA4HDIG4YQER5OGQ, seg_pos={0 0}, upload=494352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: read tcp 188.166.225.170:51206->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Dec 29 03:19:36.678583008 (563h50m36.549666043s) [noise=true , address=45.91.116.8:4045 , piece_id=R6YX47QZWG5XANY4WCPDIJC4UBX4WZT2YINYHA4HDIG4YQER5OGQ, seg_pos={0 0}, upload=494352]: Long tail cancellation due to optimal threshold Dec 29 03:19:36.678599880 (563h50m36.549682915s) [seg_pos={0 0}, upload=494352]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:19:36.678665305 (563h50m36.54974834s) [upload=494352]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:19:36.678671246 (563h50m36.549754281s) [upload=494352]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:19:36.965146340 (563h50m36.836229375s) [upload=494352]: Done uploading segments... Dec 29 03:19:36.965164043 (563h50m36.836247088s) [upload=494352]: Upload finished. err: <nil> Duration: 5.900945107s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........................................................XX.] duration: 3.42510403s trace: https://jaeger.dev.storj.io/trace/481fdf974580574a FAIL (2025-12-29 03:21:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.425132766s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X.................................................X.] duration: 37.828731253s trace: https://jaeger.dev.storj.io/trace/1584175ba8dba7fe FAIL (2025-12-29 03:21:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ece... Dec 29 03:20:05.592697598 (563h51m5.463780643s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593249339 (563h51m5.464332384s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593687443 (563h51m5.464770488s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594173598 (563h51m5.465256633s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594639957 (563h51m5.465723002s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595331994 (563h51m5.466415039s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595834151 (563h51m5.466917206s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.596329844 (563h51m5.467412899s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599128643 (563h51m5.470211688s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599196953 (563h51m5.470279988s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.601094367 (563h51m5.472177412s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.602868896 (563h51m5.473951941s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.604661328 (563h51m5.475744374s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.606462138 (563h51m5.477545183s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.609580124 (563h51m5.480663169s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.611644807 (563h51m5.482727852s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:06.901416710 (563h51m6.772499755s) [upload=494362]: Next returned nil segment Dec 29 03:20:06.901434123 (563h51m6.772517168s) [upload=494362]: Waiting for error group managing segments... Dec 29 03:20:07.237383793 (563h51m7.108466848s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.439111607 (563h51m7.310194652s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.462301153 (563h51m7.333384198s) [noise=true , address=145.79.227.21:4498 , piece_id=RTK3PTWGWTIQSNCGREQB37OID2RGK37QWEKTDYFQKASGH6SZCURA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.479891718 (563h51m7.350974763s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.486062297 (563h51m7.357145342s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.578667930 (563h51m7.449750975s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.701241555 (563h51m7.5723246s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.745868671 (563h51m7.616951706s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.799936903 (563h51m7.671019948s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.430929511 (563h51m8.302012556s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.432106221 (563h51m8.303189256s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.447633778 (563h51m8.318716813s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.459472520 (563h51m8.330555555s) [noise=true , address=145.79.227.34:4916 , piece_id=4U67LINR5A4GR2NIUVMB2IPTT5DJQOYHM6HDYKZEICKF5JY53VFA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499215165 (563h51m8.37029821s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499245653 (563h51m8.370328698s) [seg_pos={0 0}, upload=494362]: Segment reached optimal threshold of 14 pieces. Dec 29 03:20:08.499544722 (563h51m8.370627797s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.499555243 (563h51m8.370638288s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.499833272 (563h51m8.370916327s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection Dec 29 03:20:08.499843291 (563h51m8.370926366s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500107964 (563h51m8.371191019s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.500166165 (563h51m8.37124921s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500868092 (563h51m8.371951147s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53): protocol: context canceled; context canceled Dec 29 03:20:08.500880466 (563h51m8.371963511s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500945178 (563h51m8.372028223s) [seg_pos={0 0}, upload=494362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:20:08.501128358 (563h51m8.372211413s) [upload=494362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:20:08.501156331 (563h51m8.372239376s) [upload=494362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:20:08.795508553 (563h51m8.666591598s) [upload=494362]: Done uploading segments... Dec 29 03:20:08.795532639 (563h51m8.666615684s) [upload=494362]: Upload finished. err: <nil> Duration: 37.828797981s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.X..XX..X.X.X....X.X.XX..XX.....XX.X.....X.X..XXXX..XXX.XX..] duration: 5.585232019s trace: https://jaeger.dev.storj.io/trace/6995403b007d9527 FAIL (2025-12-29 03:21:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: D4GQ2INR5NU7CWIDXRGQXJSU5VIQCW5TEVOCLZHIPPJRT5N6IIQQ) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: NUPN6QWVHXWQEIAS6S2FJZZ2O7MTMODZBGIDJE5IJH4C6SLQQE5Q) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: GENODWOJOMCNHGVLBENXTAEM5SSIQTJLCJ5KJY7WNYW4V7CJARRA) context deadline exceeded ======LOGS====== Dec 29 03:19:34.142993454 (563h50m34.01407649s) [upload=494352]: Starting upload Dec 29 03:19:34.143062005 (563h50m34.01414505s) [upload=494352]: Done starting upload Dec 29 03:19:34.143139853 (563h50m34.014222888s) [upload=494352]: Uploading segments... Dec 29 03:19:34.143244613 (563h50m34.014327648s) [upload=494352]: Got next segment. Inline: false Dec 29 03:19:34.143272265 (563h50m34.0143553s) [upload=494352]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:19:34.143276754 (563h50m34.014359789s) [upload=494352]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:19:34.387054085 (563h50m34.25813713s) [seg_pos={0 0}, upload=494352]: Begin upload segment... Dec 29 03:19:34.387109671 (563h50m34.258192716s) [seg_pos={0 0}, upload=494352]: Done begin upload segment. Dec 29 03:19:34.387143886 (563h50m34.258226921s) [noise=true , address=145.79.227.42:4122 , piece_id=SIWOCNVJLLJXYBGNAHX36SHGDEBJE6HQKMQIGBLRQ4BAGWLCU75A, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.387305193 (563h50m34.258388228s) [noise=true , address=145.79.227.51:4411 , piece_id=NVEARSYSZ2WX2CVMC2MUD6QNLWGIQE3OU74E6OOM654X3OHQWJ6Q, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.388094737 (563h50m34.259177812s) [noise=true , address=145.79.227.35:4481 , piece_id=Q5SC5ODDEY5WIG5PPXZLJDDUAKGMURXU3VNNDUBR4A5ID4JJNZHA, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.388604898 (563h50m34.259687944s) [noise=true , address=45.91.116.8:4045 , piece_id=R6YX47QZWG5XANY4WCPDIJC4UBX4WZT2YINYHA4HDIG4YQER5OGQ, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.389037642 (563h50m34.260120687s) [noise=true , address=145.79.227.38:4978 , piece_id=J7RLKPC73H65EQBHJV6NQZ3RUQSFR4NF5HUH6LRQEMWBGKEUT47A, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.389458624 (563h50m34.260541669s) [noise=true , address=77.90.31.14:4663 , piece_id=D4GQ2INR5NU7CWIDXRGQXJSU5VIQCW5TEVOCLZHIPPJRT5N6IIQQ, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.389731343 (563h50m34.260814378s) [noise=true , address=77.90.31.19:4842 , piece_id=NG4YNMOQVZCPOPSXBTTNFCNP5T6CIQV2XM36IEIFOXAGCTCOWW2Q, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.390038568 (563h50m34.261121603s) [noise=true , address=77.90.31.11:4555 , piece_id=NUPN6QWVHXWQEIAS6S2FJZZ2O7MTMODZBGIDJE5IJH4C6SLQQE5Q, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.390283254 (563h50m34.261366289s) [noise=true , address=145.79.227.12:4765 , piece_id=GHZ4KSLRHU5YBSP4AJHEHWHPUSJ6RW7RCXFE7IEJWN7XTKRMVLIA, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.390322789 (563h50m34.261405824s) [noise=true , address=145.79.227.46:4602 , piece_id=3AZIKLUUQX54BXHAAI3E5WPAGN3D3M7PSYTMHLAWQ7FUNDQ4X6FQ, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.391090130 (563h50m34.262173165s) [noise=true , address=77.90.31.13:4629 , piece_id=GENODWOJOMCNHGVLBENXTAEM5SSIQTJLCJ5KJY7WNYW4V7CJARRA, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.391634356 (563h50m34.262717391s) [noise=true , address=145.79.227.26:4654 , piece_id=OQDDMZC3KYSFULPJ3KIGFA5LLRQVGT3RACMVSOL26RA3V6KXMJRA, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.392876762 (563h50m34.263959807s) [noise=true , address=145.79.227.48:4303 , piece_id=QEHTV45MB2WYGPFNKDA5PV5VE7YMOJTPXT2BEYJU37BLNIOKCBGQ, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.393487676 (563h50m34.264570711s) [noise=true , address=145.79.227.41:4085 , piece_id=BBVGVUIAMGEAI3GFJFFCSJARSFLA66BDU5B6FPOAJ4NWSDL57O6Q, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.393869944 (563h50m34.264952979s) [noise=true , address=145.79.227.50:4375 , piece_id=GES7FXSCHCU4Q6RUJD5AASHHBAVMILVZPCSS4OVITLGNV3HTKVWQ, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.394198078 (563h50m34.265281113s) [noise=true , address=145.79.227.21:4473 , piece_id=TUNQXCRHGCQODQRDHSO7DII2HS4ANYIV3LXUEL7HOPCKNZ3XQ64Q, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:36.076158618 (563h50m35.947241653s) [upload=494352]: Next returned nil segment Dec 29 03:19:36.076175882 (563h50m35.947280597s) [upload=494352]: Waiting for error group managing segments... Dec 29 03:19:36.370782238 (563h50m36.241865283s) [noise=true , address=145.79.227.46:4602 , piece_id=3AZIKLUUQX54BXHAAI3E5WPAGN3D3M7PSYTMHLAWQ7FUNDQ4X6FQ, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.372543873 (563h50m36.243626908s) [noise=true , address=145.79.227.50:4375 , piece_id=GES7FXSCHCU4Q6RUJD5AASHHBAVMILVZPCSS4OVITLGNV3HTKVWQ, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.373146620 (563h50m36.244229655s) [noise=true , address=145.79.227.42:4122 , piece_id=SIWOCNVJLLJXYBGNAHX36SHGDEBJE6HQKMQIGBLRQ4BAGWLCU75A, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.379586383 (563h50m36.250669428s) [noise=true , address=145.79.227.38:4978 , piece_id=J7RLKPC73H65EQBHJV6NQZ3RUQSFR4NF5HUH6LRQEMWBGKEUT47A, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.395642285 (563h50m36.26672532s) [noise=true , address=145.79.227.12:4765 , piece_id=GHZ4KSLRHU5YBSP4AJHEHWHPUSJ6RW7RCXFE7IEJWN7XTKRMVLIA, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.397372199 (563h50m36.268455244s) [noise=true , address=145.79.227.48:4303 , piece_id=QEHTV45MB2WYGPFNKDA5PV5VE7YMOJTPXT2BEYJU37BLNIOKCBGQ, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.399787488 (563h50m36.270870533s) [noise=true , address=145.79.227.21:4473 , piece_id=TUNQXCRHGCQODQRDHSO7DII2HS4ANYIV3LXUEL7HOPCKNZ3XQ64Q, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.403227939 (563h50m36.274310974s) [noise=true , address=145.79.227.41:4085 , piece_id=BBVGVUIAMGEAI3GFJFFCSJARSFLA66BDU5B6FPOAJ4NWSDL57O6Q, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.411425488 (563h50m36.282508533s) [noise=true , address=145.79.227.51:4411 , piece_id=NVEARSYSZ2WX2CVMC2MUD6QNLWGIQE3OU74E6OOM654X3OHQWJ6Q, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.430446266 (563h50m36.301529311s) [noise=true , address=145.79.227.26:4654 , piece_id=OQDDMZC3KYSFULPJ3KIGFA5LLRQVGT3RACMVSOL26RA3V6KXMJRA, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.640650540 (563h50m36.511733575s) [noise=true , address=77.90.31.11:4555 , piece_id=NUPN6QWVHXWQEIAS6S2FJZZ2O7MTMODZBGIDJE5IJH4C6SLQQE5Q, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.663401080 (563h50m36.534484115s) [noise=true , address=77.90.31.13:4629 , piece_id=GENODWOJOMCNHGVLBENXTAEM5SSIQTJLCJ5KJY7WNYW4V7CJARRA, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.678007813 (563h50m36.549090848s) [noise=true , address=77.90.31.14:4663 , piece_id=D4GQ2INR5NU7CWIDXRGQXJSU5VIQCW5TEVOCLZHIPPJRT5N6IIQQ, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.678026729 (563h50m36.549109764s) [seg_pos={0 0}, upload=494352]: Segment reached optimal threshold of 13 pieces. Dec 29 03:19:36.678220327 (563h50m36.549303363s) [noise=true , address=145.79.227.35:4481 , piece_id=Q5SC5ODDEY5WIG5PPXZLJDDUAKGMURXU3VNNDUBR4A5ID4JJNZHA, seg_pos={0 0}, upload=494352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 29 03:19:36.678228783 (563h50m36.549311818s) [noise=true , address=145.79.227.35:4481 , piece_id=Q5SC5ODDEY5WIG5PPXZLJDDUAKGMURXU3VNNDUBR4A5ID4JJNZHA, seg_pos={0 0}, upload=494352]: Long tail cancellation due to optimal threshold Dec 29 03:19:36.678335476 (563h50m36.549418522s) [noise=true , address=77.90.31.19:4842 , piece_id=NG4YNMOQVZCPOPSXBTTNFCNP5T6CIQV2XM36IEIFOXAGCTCOWW2Q, seg_pos={0 0}, upload=494352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 29 03:19:36.678340797 (563h50m36.549423842s) [noise=true , address=77.90.31.19:4842 , piece_id=NG4YNMOQVZCPOPSXBTTNFCNP5T6CIQV2XM36IEIFOXAGCTCOWW2Q, seg_pos={0 0}, upload=494352]: Long tail cancellation due to optimal threshold Dec 29 03:19:36.678576005 (563h50m36.54965904s) [noise=true , address=45.91.116.8:4045 , piece_id=R6YX47QZWG5XANY4WCPDIJC4UBX4WZT2YINYHA4HDIG4YQER5OGQ, seg_pos={0 0}, upload=494352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: read tcp 188.166.225.170:51206->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Dec 29 03:19:36.678583008 (563h50m36.549666043s) [noise=true , address=45.91.116.8:4045 , piece_id=R6YX47QZWG5XANY4WCPDIJC4UBX4WZT2YINYHA4HDIG4YQER5OGQ, seg_pos={0 0}, upload=494352]: Long tail cancellation due to optimal threshold Dec 29 03:19:36.678599880 (563h50m36.549682915s) [seg_pos={0 0}, upload=494352]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:19:36.678665305 (563h50m36.54974834s) [upload=494352]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:19:36.678671246 (563h50m36.549754281s) [upload=494352]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:19:36.965146340 (563h50m36.836229375s) [upload=494352]: Done uploading segments... Dec 29 03:19:36.965164043 (563h50m36.836247088s) [upload=494352]: Upload finished. err: <nil> Duration: 5.585265344s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X.................................................X] duration: 1m0.006178921s trace: https://jaeger.dev.storj.io/trace/1584175ba8dba7fe FAIL (2025-12-29 03:21:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ece... Dec 29 03:20:05.592697598 (563h51m5.463780643s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593249339 (563h51m5.464332384s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.593687443 (563h51m5.464770488s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594173598 (563h51m5.465256633s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.594639957 (563h51m5.465723002s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595331994 (563h51m5.466415039s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.595834151 (563h51m5.466917206s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.596329844 (563h51m5.467412899s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599128643 (563h51m5.470211688s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.599196953 (563h51m5.470279988s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.601094367 (563h51m5.472177412s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.602868896 (563h51m5.473951941s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.604661328 (563h51m5.475744374s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.606462138 (563h51m5.477545183s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.609580124 (563h51m5.480663169s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:05.611644807 (563h51m5.482727852s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Uploading piece... Dec 29 03:20:06.901416710 (563h51m6.772499755s) [upload=494362]: Next returned nil segment Dec 29 03:20:06.901434123 (563h51m6.772517168s) [upload=494362]: Waiting for error group managing segments... Dec 29 03:20:07.237383793 (563h51m7.108466848s) [noise=true , address=64.49.13.140:4515 , piece_id=BB4RND6DQPMXBJHNTIFIQWVZOFZFHIFB6KBYEUOKOLJFDCWRIMJA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.439111607 (563h51m7.310194652s) [noise=true , address=145.79.227.9:4382 , piece_id=WQDVG2KXSEDZHJJOT2ZFSAPQXVLJZXMO5HQXXEU6XJ7U7CAQWZVQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.462301153 (563h51m7.333384198s) [noise=true , address=145.79.227.21:4498 , piece_id=RTK3PTWGWTIQSNCGREQB37OID2RGK37QWEKTDYFQKASGH6SZCURA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.479891718 (563h51m7.350974763s) [noise=true , address=145.79.227.12:4778 , piece_id=KHBVVV3WSILPU4CQK4HAUC37D75DUWGOJWZFY53SBDLVY3F3EBAA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.486062297 (563h51m7.357145342s) [noise=true , address=145.79.227.2:4142 , piece_id=GS7XTVLRWWXMZKGYSVDGKAGR6QX3OJUTK4TNNVPPA26XS5KSHKBQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.578667930 (563h51m7.449750975s) [noise=true , address=145.79.227.50:4391 , piece_id=GX6GJKCSCDKGMYRAPNP4Z347JJMKPWFLDT2QGOSSZGAYKNZNTDHA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.701241555 (563h51m7.5723246s) [noise=true , address=45.91.116.13:4177 , piece_id=YIJBXUI5PGKEJOY3USREMMV35V2VMXXFILDOUNJPXMIHVD5PZIUA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.745868671 (563h51m7.616951706s) [noise=true , address=77.90.31.13:4644 , piece_id=VJZEITJXVPJHT6FPHXEFDMWDGR4XQNAZMDGP7BOKEQEAKKN5LKJQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:07.799936903 (563h51m7.671019948s) [noise=true , address=77.90.31.16:4739 , piece_id=5I3LCPAI2IF5Z5MZWQNKDM7LKBYEYIPE3EBUAD6DOLBPW34FDNPA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.430929511 (563h51m8.302012556s) [noise=true , address=77.90.31.11:4588 , piece_id=XU7QUQZHRH27X64OR4NXJERPPIHCPBEGL7POHUL5Z32LTFMI5MKQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.432106221 (563h51m8.303189256s) [noise=true , address=145.79.227.35:4493 , piece_id=A2ZOH3MMXV5S7JPOWOCQZMYQEMF3SJHTYIIOB2656AVAXP24UAYA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.447633778 (563h51m8.318716813s) [noise=true , address=145.79.227.49:4355 , piece_id=ZBF32JIHB33TYP2KB427V7OQRRNXP2ZOTLZNFRQ5XUH7HGPG2RTA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.459472520 (563h51m8.330555555s) [noise=true , address=145.79.227.34:4916 , piece_id=4U67LINR5A4GR2NIUVMB2IPTT5DJQOYHM6HDYKZEICKF5JY53VFA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499215165 (563h51m8.37029821s) [noise=true , address=145.79.227.43:4170 , piece_id=F7H3IF2SNNGTIV2CFKXB3GIA6TGRLPSOLA6TPBUW5FQVWULEK4GQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: <nil> Dec 29 03:20:08.499245653 (563h51m8.370328698s) [seg_pos={0 0}, upload=494362]: Segment reached optimal threshold of 14 pieces. Dec 29 03:20:08.499544722 (563h51m8.370627797s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.499555243 (563h51m8.370638288s) [noise=true , address=145.79.227.35:4507 , piece_id=SNS3BVKFWUKQYMNZCWMOHH563GXNM5IIB2QMLGVCCMIXHEV7MJOA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.499833272 (563h51m8.370916327s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY): protocol: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39578->77.90.31.18:4838: use of closed network connection Dec 29 03:20:08.499843291 (563h51m8.370926366s) [noise=true , address=77.90.31.18:4838 , piece_id=TP42RP5AUZSP6LNKK7ON4FMDFVF6GRX3FLRYWBS56SCKFAZNPQSA, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500107964 (563h51m8.371191019s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6): protocol: expected piece hash; context canceled; EOF Dec 29 03:20:08.500166165 (563h51m8.37124921s) [noise=true , address=77.90.31.21:4909 , piece_id=KKTXOQ647SWOMJ7J4DC6CMHBSI65JJD2C5IMZZWI3ZSQI7SXLY2A, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500868092 (563h51m8.371951147s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V4Dj8k4kAnrS6GfSJvvDDPvyHTiJziufBCWWjNoPe1Yvc6M53): protocol: context canceled; context canceled Dec 29 03:20:08.500880466 (563h51m8.371963511s) [noise=true , address=77.90.31.12:4605 , piece_id=CEYXIJYUE4MGBAOGMKOWIQZBRFRC23FFUZCP5AW7N4MCT76B5DWQ, seg_pos={0 0}, upload=494362]: Long tail cancellation due to optimal threshold Dec 29 03:20:08.500945178 (563h51m8.372028223s) [seg_pos={0 0}, upload=494362]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 03:20:08.501128358 (563h51m8.372211413s) [upload=494362]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:20:08.501156331 (563h51m8.372239376s) [upload=494362]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:20:08.795508553 (563h51m8.666591598s) [upload=494362]: Done uploading segments... Dec 29 03:20:08.795532639 (563h51m8.666615684s) [upload=494362]: Upload finished. err: <nil> Duration: 1m0.006192246s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................................................XX] duration: 5.001861899s trace: https://jaeger.dev.storj.io/trace/481fdf974580574a FAIL (2025-12-29 03:20:55 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001891254s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [X.X..XX..X.X.X....X.X.XX..XX.....XX.X.....X.X..XXXX..XXX.XX.] duration: 6.744566693s trace: https://jaeger.dev.storj.io/trace/6995403b007d9527 FAIL (2025-12-29 03:20:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: D4GQ2INR5NU7CWIDXRGQXJSU5VIQCW5TEVOCLZHIPPJRT5N6IIQQ) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: NUPN6QWVHXWQEIAS6S2FJZZ2O7MTMODZBGIDJE5IJH4C6SLQQE5Q) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: GENODWOJOMCNHGVLBENXTAEM5SSIQTJLCJ5KJY7WNYW4V7CJARRA) context deadline exceeded ======LOGS====== Dec 29 03:19:34.142993454 (563h50m34.01407649s) [upload=494352]: Starting upload Dec 29 03:19:34.143062005 (563h50m34.01414505s) [upload=494352]: Done starting upload Dec 29 03:19:34.143139853 (563h50m34.014222888s) [upload=494352]: Uploading segments... Dec 29 03:19:34.143244613 (563h50m34.014327648s) [upload=494352]: Got next segment. Inline: false Dec 29 03:19:34.143272265 (563h50m34.0143553s) [upload=494352]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:19:34.143276754 (563h50m34.014359789s) [upload=494352]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:19:34.387054085 (563h50m34.25813713s) [seg_pos={0 0}, upload=494352]: Begin upload segment... Dec 29 03:19:34.387109671 (563h50m34.258192716s) [seg_pos={0 0}, upload=494352]: Done begin upload segment. Dec 29 03:19:34.387143886 (563h50m34.258226921s) [noise=true , address=145.79.227.42:4122 , piece_id=SIWOCNVJLLJXYBGNAHX36SHGDEBJE6HQKMQIGBLRQ4BAGWLCU75A, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.387305193 (563h50m34.258388228s) [noise=true , address=145.79.227.51:4411 , piece_id=NVEARSYSZ2WX2CVMC2MUD6QNLWGIQE3OU74E6OOM654X3OHQWJ6Q, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.388094737 (563h50m34.259177812s) [noise=true , address=145.79.227.35:4481 , piece_id=Q5SC5ODDEY5WIG5PPXZLJDDUAKGMURXU3VNNDUBR4A5ID4JJNZHA, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.388604898 (563h50m34.259687944s) [noise=true , address=45.91.116.8:4045 , piece_id=R6YX47QZWG5XANY4WCPDIJC4UBX4WZT2YINYHA4HDIG4YQER5OGQ, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.389037642 (563h50m34.260120687s) [noise=true , address=145.79.227.38:4978 , piece_id=J7RLKPC73H65EQBHJV6NQZ3RUQSFR4NF5HUH6LRQEMWBGKEUT47A, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.389458624 (563h50m34.260541669s) [noise=true , address=77.90.31.14:4663 , piece_id=D4GQ2INR5NU7CWIDXRGQXJSU5VIQCW5TEVOCLZHIPPJRT5N6IIQQ, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.389731343 (563h50m34.260814378s) [noise=true , address=77.90.31.19:4842 , piece_id=NG4YNMOQVZCPOPSXBTTNFCNP5T6CIQV2XM36IEIFOXAGCTCOWW2Q, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.390038568 (563h50m34.261121603s) [noise=true , address=77.90.31.11:4555 , piece_id=NUPN6QWVHXWQEIAS6S2FJZZ2O7MTMODZBGIDJE5IJH4C6SLQQE5Q, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.390283254 (563h50m34.261366289s) [noise=true , address=145.79.227.12:4765 , piece_id=GHZ4KSLRHU5YBSP4AJHEHWHPUSJ6RW7RCXFE7IEJWN7XTKRMVLIA, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.390322789 (563h50m34.261405824s) [noise=true , address=145.79.227.46:4602 , piece_id=3AZIKLUUQX54BXHAAI3E5WPAGN3D3M7PSYTMHLAWQ7FUNDQ4X6FQ, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.391090130 (563h50m34.262173165s) [noise=true , address=77.90.31.13:4629 , piece_id=GENODWOJOMCNHGVLBENXTAEM5SSIQTJLCJ5KJY7WNYW4V7CJARRA, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.391634356 (563h50m34.262717391s) [noise=true , address=145.79.227.26:4654 , piece_id=OQDDMZC3KYSFULPJ3KIGFA5LLRQVGT3RACMVSOL26RA3V6KXMJRA, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.392876762 (563h50m34.263959807s) [noise=true , address=145.79.227.48:4303 , piece_id=QEHTV45MB2WYGPFNKDA5PV5VE7YMOJTPXT2BEYJU37BLNIOKCBGQ, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.393487676 (563h50m34.264570711s) [noise=true , address=145.79.227.41:4085 , piece_id=BBVGVUIAMGEAI3GFJFFCSJARSFLA66BDU5B6FPOAJ4NWSDL57O6Q, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.393869944 (563h50m34.264952979s) [noise=true , address=145.79.227.50:4375 , piece_id=GES7FXSCHCU4Q6RUJD5AASHHBAVMILVZPCSS4OVITLGNV3HTKVWQ, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.394198078 (563h50m34.265281113s) [noise=true , address=145.79.227.21:4473 , piece_id=TUNQXCRHGCQODQRDHSO7DII2HS4ANYIV3LXUEL7HOPCKNZ3XQ64Q, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:36.076158618 (563h50m35.947241653s) [upload=494352]: Next returned nil segment Dec 29 03:19:36.076175882 (563h50m35.947280597s) [upload=494352]: Waiting for error group managing segments... Dec 29 03:19:36.370782238 (563h50m36.241865283s) [noise=true , address=145.79.227.46:4602 , piece_id=3AZIKLUUQX54BXHAAI3E5WPAGN3D3M7PSYTMHLAWQ7FUNDQ4X6FQ, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.372543873 (563h50m36.243626908s) [noise=true , address=145.79.227.50:4375 , piece_id=GES7FXSCHCU4Q6RUJD5AASHHBAVMILVZPCSS4OVITLGNV3HTKVWQ, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.373146620 (563h50m36.244229655s) [noise=true , address=145.79.227.42:4122 , piece_id=SIWOCNVJLLJXYBGNAHX36SHGDEBJE6HQKMQIGBLRQ4BAGWLCU75A, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.379586383 (563h50m36.250669428s) [noise=true , address=145.79.227.38:4978 , piece_id=J7RLKPC73H65EQBHJV6NQZ3RUQSFR4NF5HUH6LRQEMWBGKEUT47A, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.395642285 (563h50m36.26672532s) [noise=true , address=145.79.227.12:4765 , piece_id=GHZ4KSLRHU5YBSP4AJHEHWHPUSJ6RW7RCXFE7IEJWN7XTKRMVLIA, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.397372199 (563h50m36.268455244s) [noise=true , address=145.79.227.48:4303 , piece_id=QEHTV45MB2WYGPFNKDA5PV5VE7YMOJTPXT2BEYJU37BLNIOKCBGQ, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.399787488 (563h50m36.270870533s) [noise=true , address=145.79.227.21:4473 , piece_id=TUNQXCRHGCQODQRDHSO7DII2HS4ANYIV3LXUEL7HOPCKNZ3XQ64Q, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.403227939 (563h50m36.274310974s) [noise=true , address=145.79.227.41:4085 , piece_id=BBVGVUIAMGEAI3GFJFFCSJARSFLA66BDU5B6FPOAJ4NWSDL57O6Q, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.411425488 (563h50m36.282508533s) [noise=true , address=145.79.227.51:4411 , piece_id=NVEARSYSZ2WX2CVMC2MUD6QNLWGIQE3OU74E6OOM654X3OHQWJ6Q, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.430446266 (563h50m36.301529311s) [noise=true , address=145.79.227.26:4654 , piece_id=OQDDMZC3KYSFULPJ3KIGFA5LLRQVGT3RACMVSOL26RA3V6KXMJRA, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.640650540 (563h50m36.511733575s) [noise=true , address=77.90.31.11:4555 , piece_id=NUPN6QWVHXWQEIAS6S2FJZZ2O7MTMODZBGIDJE5IJH4C6SLQQE5Q, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.663401080 (563h50m36.534484115s) [noise=true , address=77.90.31.13:4629 , piece_id=GENODWOJOMCNHGVLBENXTAEM5SSIQTJLCJ5KJY7WNYW4V7CJARRA, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.678007813 (563h50m36.549090848s) [noise=true , address=77.90.31.14:4663 , piece_id=D4GQ2INR5NU7CWIDXRGQXJSU5VIQCW5TEVOCLZHIPPJRT5N6IIQQ, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.678026729 (563h50m36.549109764s) [seg_pos={0 0}, upload=494352]: Segment reached optimal threshold of 13 pieces. Dec 29 03:19:36.678220327 (563h50m36.549303363s) [noise=true , address=145.79.227.35:4481 , piece_id=Q5SC5ODDEY5WIG5PPXZLJDDUAKGMURXU3VNNDUBR4A5ID4JJNZHA, seg_pos={0 0}, upload=494352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 29 03:19:36.678228783 (563h50m36.549311818s) [noise=true , address=145.79.227.35:4481 , piece_id=Q5SC5ODDEY5WIG5PPXZLJDDUAKGMURXU3VNNDUBR4A5ID4JJNZHA, seg_pos={0 0}, upload=494352]: Long tail cancellation due to optimal threshold Dec 29 03:19:36.678335476 (563h50m36.549418522s) [noise=true , address=77.90.31.19:4842 , piece_id=NG4YNMOQVZCPOPSXBTTNFCNP5T6CIQV2XM36IEIFOXAGCTCOWW2Q, seg_pos={0 0}, upload=494352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 29 03:19:36.678340797 (563h50m36.549423842s) [noise=true , address=77.90.31.19:4842 , piece_id=NG4YNMOQVZCPOPSXBTTNFCNP5T6CIQV2XM36IEIFOXAGCTCOWW2Q, seg_pos={0 0}, upload=494352]: Long tail cancellation due to optimal threshold Dec 29 03:19:36.678576005 (563h50m36.54965904s) [noise=true , address=45.91.116.8:4045 , piece_id=R6YX47QZWG5XANY4WCPDIJC4UBX4WZT2YINYHA4HDIG4YQER5OGQ, seg_pos={0 0}, upload=494352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: read tcp 188.166.225.170:51206->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Dec 29 03:19:36.678583008 (563h50m36.549666043s) [noise=true , address=45.91.116.8:4045 , piece_id=R6YX47QZWG5XANY4WCPDIJC4UBX4WZT2YINYHA4HDIG4YQER5OGQ, seg_pos={0 0}, upload=494352]: Long tail cancellation due to optimal threshold Dec 29 03:19:36.678599880 (563h50m36.549682915s) [seg_pos={0 0}, upload=494352]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:19:36.678665305 (563h50m36.54974834s) [upload=494352]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:19:36.678671246 (563h50m36.549754281s) [upload=494352]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:19:36.965146340 (563h50m36.836229375s) [upload=494352]: Done uploading segments... Dec 29 03:19:36.965164043 (563h50m36.836247088s) [upload=494352]: Upload finished. err: <nil> Duration: 6.744608552s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 29.217477881s trace: https://jaeger.dev.storj.io/trace/4971cdf1bfe7285d FAIL (2025-12-29 03:20:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oise=true , address=145.79.227.40:4071 , piece_id=4UFHNYWEFPHGBNJ5AWFRHBLOJD32DCUF6ITMEBILQEQIAL3JZYRA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.692976565 (563h11m19.56405996s) [noise=true , address=145.79.227.34:4936 , piece_id=RVZM43EUJLI3EC37SZ5VZSWEIPQBVTOTMTZCHGI5SITCNYC3R5JQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.693422042 (563h11m19.564505087s) [noise=true , address=45.91.116.8:4065 , piece_id=IOYN7IQWNZYDU3FQDUQT7JUGJMVCGSMWDT54JF2YC345WBGT36VA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.693937634 (563h11m19.565020669s) [noise=true , address=145.79.227.44:4217 , piece_id=EPL7C7MCL5AAODIXS57QYAR2DM6CBBQLCYRCS2JN2D6L67EW54TQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.693954627 (563h11m19.565037832s) [noise=true , address=145.79.227.49:4349 , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.694295767 (563h11m19.565378812s) [noise=true , address=145.79.227.5:4248 , piece_id=4QTK7ABVI2LYKUX6XAR5SXJS6MFJLIDQBZCZLIOK5KP2J7IFBBLQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.694391017 (563h11m19.565474052s) [noise=true , address=145.79.227.35:4504 , piece_id=WEIQGYBGBME6332Z7JYRECH3UME4VDO6T5HLKYDPJR3R33AO76PA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.694589466 (563h11m19.565672501s) [noise=true , address=145.79.227.43:4161 , piece_id=FE32SHJFWMZBIAGLNPSU4CNLTHY5WM2WLG5LHSYSCGBB7MEP2FHA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.694983245 (563h11m19.5660663s) [noise=true , address=45.91.116.5:4324 , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.695183036 (563h11m19.566266071s) [noise=true , address=145.79.227.23:4553 , piece_id=36U6CKVCCKPVQPDUIZK4KSVX67SHIRMGQPL7DNWLY5EXQSY6RWRQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.695537310 (563h11m19.566620345s) [noise=true , address=5.231.73.3:4011 , piece_id=O4Y5KM6P5F47HALP5YRCBKL6KA5TLPRJNO4CV7B3SB6AU7MKAUVQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.695839536 (563h11m19.566922581s) [noise=true , address=145.79.227.25:4620 , piece_id=ZXO642URK4B6NCEVWZWBNJBE3IG6Z5J72WLJ2CEDQJXZGHZC2PAA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.696306775 (563h11m19.56738981s) [noise=true , address=145.79.227.4:4218 , piece_id=63JGO44NUSJXOQKBOIQWXIFGVZYJYQXZWHRV4B5COGFQ24GA6SXA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.696687711 (563h11m19.567770746s) [noise=true , address=77.90.31.11:4569 , piece_id=6VEL6XUEF7VJBXDJUB5YKHUIXH3VE32JX6AKILU55NBOLELMVMAQ, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.697126757 (563h11m19.568209792s) [noise=true , address=77.90.31.15:4723 , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:19.697988819 (563h11m19.569071864s) [noise=true , address=145.79.227.30:4775 , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Uploading piece... Dec 29 02:40:21.205435522 (563h11m21.076518557s) [upload=493768]: Next returned nil segment Dec 29 02:40:21.205450580 (563h11m21.076533625s) [upload=493768]: Waiting for error group managing segments... Dec 29 02:40:21.617703991 (563h11m21.488787026s) [noise=true , address=145.79.227.25:4620 , piece_id=ZXO642URK4B6NCEVWZWBNJBE3IG6Z5J72WLJ2CEDQJXZGHZC2PAA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.655871746 (563h11m21.526954792s) [noise=true , address=145.79.227.5:4248 , piece_id=4QTK7ABVI2LYKUX6XAR5SXJS6MFJLIDQBZCZLIOK5KP2J7IFBBLQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.656078530 (563h11m21.527161565s) [noise=true , address=145.79.227.4:4218 , piece_id=63JGO44NUSJXOQKBOIQWXIFGVZYJYQXZWHRV4B5COGFQ24GA6SXA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.681311167 (563h11m21.552394212s) [noise=true , address=145.79.227.23:4553 , piece_id=36U6CKVCCKPVQPDUIZK4KSVX67SHIRMGQPL7DNWLY5EXQSY6RWRQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.685428146 (563h11m21.556511211s) [noise=true , address=5.231.73.3:4011 , piece_id=O4Y5KM6P5F47HALP5YRCBKL6KA5TLPRJNO4CV7B3SB6AU7MKAUVQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.793760553 (563h11m21.664843599s) [noise=true , address=45.91.116.6:4990 , piece_id=IX7D4CRUNF3KXZLPRRPUCDLAAJAK2SKQ5FUU5P2M7JBJHE7WQO6Q, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.814631916 (563h11m21.685714961s) [noise=true , address=45.91.116.8:4065 , piece_id=IOYN7IQWNZYDU3FQDUQT7JUGJMVCGSMWDT54JF2YC345WBGT36VA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:21.831572271 (563h11m21.702655306s) [noise=true , address=77.90.31.11:4569 , piece_id=6VEL6XUEF7VJBXDJUB5YKHUIXH3VE32JX6AKILU55NBOLELMVMAQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.504028712 (563h11m22.375111757s) [noise=true , address=145.79.227.40:4071 , piece_id=4UFHNYWEFPHGBNJ5AWFRHBLOJD32DCUF6ITMEBILQEQIAL3JZYRA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.516925939 (563h11m22.388008974s) [noise=true , address=77.90.31.21:4885 , piece_id=WJDRV5HTSMWMD7TQEYXCHSDCBVXMS4CC32UTCW3NS7LWATNXY3NQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.546716564 (563h11m22.417799609s) [noise=true , address=145.79.227.43:4161 , piece_id=FE32SHJFWMZBIAGLNPSU4CNLTHY5WM2WLG5LHSYSCGBB7MEP2FHA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.552365079 (563h11m22.423448124s) [noise=true , address=145.79.227.35:4504 , piece_id=WEIQGYBGBME6332Z7JYRECH3UME4VDO6T5HLKYDPJR3R33AO76PA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.553409438 (563h11m22.424492473s) [noise=true , address=145.79.227.44:4217 , piece_id=EPL7C7MCL5AAODIXS57QYAR2DM6CBBQLCYRCS2JN2D6L67EW54TQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.566950603 (563h11m22.438033638s) [noise=true , address=145.79.227.34:4936 , piece_id=RVZM43EUJLI3EC37SZ5VZSWEIPQBVTOTMTZCHGI5SITCNYC3R5JQ, seg_pos={0 0}, upload=493768]: Done uploading piece. err: <nil> Dec 29 02:40:22.566971393 (563h11m22.438054428s) [seg_pos={0 0}, upload=493768]: Segment reached optimal threshold of 14 pieces. Dec 29 02:40:22.567174610 (563h11m22.438257655s) [noise=true , address=77.90.31.15:4723 , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EGKHmd15augxZS76bCbS9f7Kzj8CQSkX1tvSCapydjVqKJseS): protocol: expected piece hash; context canceled; EOF Dec 29 02:40:22.567181653 (563h11m22.438264688s) [noise=true , address=77.90.31.15:4723 , piece_id=L54MAW7PU4FORHSKI67DIF62GKH65ISQFE3F37K4YDBUXEQZRLQA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold Dec 29 02:40:22.567422061 (563h11m22.438505096s) [noise=true , address=45.91.116.5:4324 , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12btBm53zb1MCTy6b7CUeujUruMR9nxBoRgqhHmG5ffPf7rL4F): protocol: expected piece hash; context canceled; EOF Dec 29 02:40:22.567440646 (563h11m22.438523691s) [noise=true , address=45.91.116.5:4324 , piece_id=OS4QRKAF6WGNIOWREIMTFU3SJBKFKOEKJL6O6QFORAGI7ERIYQAA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold Dec 29 02:40:22.567456967 (563h11m22.438540012s) [noise=true , address=145.79.227.30:4775 , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj): protocol: expected piece hash; context canceled; EOF Dec 29 02:40:22.567463530 (563h11m22.438546565s) [noise=true , address=145.79.227.30:4775 , piece_id=YGLHYBPTTWXNUQI73KUIJW7AR337IOU5CYMDOVKZAAU3775G4GDA, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold Dec 29 02:40:22.567539434 (563h11m22.438622469s) [noise=true , address=145.79.227.49:4349 , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CrzYmDPm95TgyE7SHJsFVbL1JUTdwMmZVge4uEkSCbeP8iBEk): protocol: write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection; write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39788->145.79.227.49:4349: use of closed network connection Dec 29 02:40:22.567547449 (563h11m22.438630484s) [noise=true , address=145.79.227.49:4349 , piece_id=VIBIWLOBLVANSG67SDLBVXOZ4BMZZJT4VVY3CNIOFJRDM2AWWM7Q, seg_pos={0 0}, upload=493768]: Long tail cancellation due to optimal threshold Dec 29 02:40:22.567566506 (563h11m22.438649541s) [seg_pos={0 0}, upload=493768]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 02:40:22.567634766 (563h11m22.438717811s) [upload=493768]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 02:40:22.567640006 (563h11m22.438723041s) [upload=493768]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 02:40:22.870735006 (563h11m22.741818041s) [upload=493768]: Done uploading segments... Dec 29 02:40:22.870755105 (563h11m22.74183814s) [upload=493768]: Upload finished. err: <nil> Duration: 29.217509653s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [.X.X..XX..X.X.X....X.X.XX..XX.....XX.X.....X.X..XXXX..XXX.XX] duration: 30.004444593s trace: https://jaeger.dev.storj.io/trace/6995403b007d9527 FAIL (2025-12-29 03:20:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: D4GQ2INR5NU7CWIDXRGQXJSU5VIQCW5TEVOCLZHIPPJRT5N6IIQQ) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: NUPN6QWVHXWQEIAS6S2FJZZ2O7MTMODZBGIDJE5IJH4C6SLQQE5Q) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: GENODWOJOMCNHGVLBENXTAEM5SSIQTJLCJ5KJY7WNYW4V7CJARRA) context deadline exceeded ======LOGS====== Dec 29 03:19:34.142993454 (563h50m34.01407649s) [upload=494352]: Starting upload Dec 29 03:19:34.143062005 (563h50m34.01414505s) [upload=494352]: Done starting upload Dec 29 03:19:34.143139853 (563h50m34.014222888s) [upload=494352]: Uploading segments... Dec 29 03:19:34.143244613 (563h50m34.014327648s) [upload=494352]: Got next segment. Inline: false Dec 29 03:19:34.143272265 (563h50m34.0143553s) [upload=494352]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 03:19:34.143276754 (563h50m34.014359789s) [upload=494352]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 03:19:34.387054085 (563h50m34.25813713s) [seg_pos={0 0}, upload=494352]: Begin upload segment... Dec 29 03:19:34.387109671 (563h50m34.258192716s) [seg_pos={0 0}, upload=494352]: Done begin upload segment. Dec 29 03:19:34.387143886 (563h50m34.258226921s) [noise=true , address=145.79.227.42:4122 , piece_id=SIWOCNVJLLJXYBGNAHX36SHGDEBJE6HQKMQIGBLRQ4BAGWLCU75A, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.387305193 (563h50m34.258388228s) [noise=true , address=145.79.227.51:4411 , piece_id=NVEARSYSZ2WX2CVMC2MUD6QNLWGIQE3OU74E6OOM654X3OHQWJ6Q, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.388094737 (563h50m34.259177812s) [noise=true , address=145.79.227.35:4481 , piece_id=Q5SC5ODDEY5WIG5PPXZLJDDUAKGMURXU3VNNDUBR4A5ID4JJNZHA, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.388604898 (563h50m34.259687944s) [noise=true , address=45.91.116.8:4045 , piece_id=R6YX47QZWG5XANY4WCPDIJC4UBX4WZT2YINYHA4HDIG4YQER5OGQ, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.389037642 (563h50m34.260120687s) [noise=true , address=145.79.227.38:4978 , piece_id=J7RLKPC73H65EQBHJV6NQZ3RUQSFR4NF5HUH6LRQEMWBGKEUT47A, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.389458624 (563h50m34.260541669s) [noise=true , address=77.90.31.14:4663 , piece_id=D4GQ2INR5NU7CWIDXRGQXJSU5VIQCW5TEVOCLZHIPPJRT5N6IIQQ, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.389731343 (563h50m34.260814378s) [noise=true , address=77.90.31.19:4842 , piece_id=NG4YNMOQVZCPOPSXBTTNFCNP5T6CIQV2XM36IEIFOXAGCTCOWW2Q, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.390038568 (563h50m34.261121603s) [noise=true , address=77.90.31.11:4555 , piece_id=NUPN6QWVHXWQEIAS6S2FJZZ2O7MTMODZBGIDJE5IJH4C6SLQQE5Q, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.390283254 (563h50m34.261366289s) [noise=true , address=145.79.227.12:4765 , piece_id=GHZ4KSLRHU5YBSP4AJHEHWHPUSJ6RW7RCXFE7IEJWN7XTKRMVLIA, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.390322789 (563h50m34.261405824s) [noise=true , address=145.79.227.46:4602 , piece_id=3AZIKLUUQX54BXHAAI3E5WPAGN3D3M7PSYTMHLAWQ7FUNDQ4X6FQ, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.391090130 (563h50m34.262173165s) [noise=true , address=77.90.31.13:4629 , piece_id=GENODWOJOMCNHGVLBENXTAEM5SSIQTJLCJ5KJY7WNYW4V7CJARRA, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.391634356 (563h50m34.262717391s) [noise=true , address=145.79.227.26:4654 , piece_id=OQDDMZC3KYSFULPJ3KIGFA5LLRQVGT3RACMVSOL26RA3V6KXMJRA, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.392876762 (563h50m34.263959807s) [noise=true , address=145.79.227.48:4303 , piece_id=QEHTV45MB2WYGPFNKDA5PV5VE7YMOJTPXT2BEYJU37BLNIOKCBGQ, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.393487676 (563h50m34.264570711s) [noise=true , address=145.79.227.41:4085 , piece_id=BBVGVUIAMGEAI3GFJFFCSJARSFLA66BDU5B6FPOAJ4NWSDL57O6Q, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.393869944 (563h50m34.264952979s) [noise=true , address=145.79.227.50:4375 , piece_id=GES7FXSCHCU4Q6RUJD5AASHHBAVMILVZPCSS4OVITLGNV3HTKVWQ, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:34.394198078 (563h50m34.265281113s) [noise=true , address=145.79.227.21:4473 , piece_id=TUNQXCRHGCQODQRDHSO7DII2HS4ANYIV3LXUEL7HOPCKNZ3XQ64Q, seg_pos={0 0}, upload=494352]: Uploading piece... Dec 29 03:19:36.076158618 (563h50m35.947241653s) [upload=494352]: Next returned nil segment Dec 29 03:19:36.076175882 (563h50m35.947280597s) [upload=494352]: Waiting for error group managing segments... Dec 29 03:19:36.370782238 (563h50m36.241865283s) [noise=true , address=145.79.227.46:4602 , piece_id=3AZIKLUUQX54BXHAAI3E5WPAGN3D3M7PSYTMHLAWQ7FUNDQ4X6FQ, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.372543873 (563h50m36.243626908s) [noise=true , address=145.79.227.50:4375 , piece_id=GES7FXSCHCU4Q6RUJD5AASHHBAVMILVZPCSS4OVITLGNV3HTKVWQ, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.373146620 (563h50m36.244229655s) [noise=true , address=145.79.227.42:4122 , piece_id=SIWOCNVJLLJXYBGNAHX36SHGDEBJE6HQKMQIGBLRQ4BAGWLCU75A, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.379586383 (563h50m36.250669428s) [noise=true , address=145.79.227.38:4978 , piece_id=J7RLKPC73H65EQBHJV6NQZ3RUQSFR4NF5HUH6LRQEMWBGKEUT47A, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.395642285 (563h50m36.26672532s) [noise=true , address=145.79.227.12:4765 , piece_id=GHZ4KSLRHU5YBSP4AJHEHWHPUSJ6RW7RCXFE7IEJWN7XTKRMVLIA, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.397372199 (563h50m36.268455244s) [noise=true , address=145.79.227.48:4303 , piece_id=QEHTV45MB2WYGPFNKDA5PV5VE7YMOJTPXT2BEYJU37BLNIOKCBGQ, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.399787488 (563h50m36.270870533s) [noise=true , address=145.79.227.21:4473 , piece_id=TUNQXCRHGCQODQRDHSO7DII2HS4ANYIV3LXUEL7HOPCKNZ3XQ64Q, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.403227939 (563h50m36.274310974s) [noise=true , address=145.79.227.41:4085 , piece_id=BBVGVUIAMGEAI3GFJFFCSJARSFLA66BDU5B6FPOAJ4NWSDL57O6Q, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.411425488 (563h50m36.282508533s) [noise=true , address=145.79.227.51:4411 , piece_id=NVEARSYSZ2WX2CVMC2MUD6QNLWGIQE3OU74E6OOM654X3OHQWJ6Q, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.430446266 (563h50m36.301529311s) [noise=true , address=145.79.227.26:4654 , piece_id=OQDDMZC3KYSFULPJ3KIGFA5LLRQVGT3RACMVSOL26RA3V6KXMJRA, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.640650540 (563h50m36.511733575s) [noise=true , address=77.90.31.11:4555 , piece_id=NUPN6QWVHXWQEIAS6S2FJZZ2O7MTMODZBGIDJE5IJH4C6SLQQE5Q, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.663401080 (563h50m36.534484115s) [noise=true , address=77.90.31.13:4629 , piece_id=GENODWOJOMCNHGVLBENXTAEM5SSIQTJLCJ5KJY7WNYW4V7CJARRA, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.678007813 (563h50m36.549090848s) [noise=true , address=77.90.31.14:4663 , piece_id=D4GQ2INR5NU7CWIDXRGQXJSU5VIQCW5TEVOCLZHIPPJRT5N6IIQQ, seg_pos={0 0}, upload=494352]: Done uploading piece. err: <nil> Dec 29 03:19:36.678026729 (563h50m36.549109764s) [seg_pos={0 0}, upload=494352]: Segment reached optimal threshold of 13 pieces. Dec 29 03:19:36.678220327 (563h50m36.549303363s) [noise=true , address=145.79.227.35:4481 , piece_id=Q5SC5ODDEY5WIG5PPXZLJDDUAKGMURXU3VNNDUBR4A5ID4JJNZHA, seg_pos={0 0}, upload=494352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 29 03:19:36.678228783 (563h50m36.549311818s) [noise=true , address=145.79.227.35:4481 , piece_id=Q5SC5ODDEY5WIG5PPXZLJDDUAKGMURXU3VNNDUBR4A5ID4JJNZHA, seg_pos={0 0}, upload=494352]: Long tail cancellation due to optimal threshold Dec 29 03:19:36.678335476 (563h50m36.549418522s) [noise=true , address=77.90.31.19:4842 , piece_id=NG4YNMOQVZCPOPSXBTTNFCNP5T6CIQV2XM36IEIFOXAGCTCOWW2Q, seg_pos={0 0}, upload=494352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 29 03:19:36.678340797 (563h50m36.549423842s) [noise=true , address=77.90.31.19:4842 , piece_id=NG4YNMOQVZCPOPSXBTTNFCNP5T6CIQV2XM36IEIFOXAGCTCOWW2Q, seg_pos={0 0}, upload=494352]: Long tail cancellation due to optimal threshold Dec 29 03:19:36.678576005 (563h50m36.54965904s) [noise=true , address=45.91.116.8:4045 , piece_id=R6YX47QZWG5XANY4WCPDIJC4UBX4WZT2YINYHA4HDIG4YQER5OGQ, seg_pos={0 0}, upload=494352]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: read tcp 188.166.225.170:51206->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled Dec 29 03:19:36.678583008 (563h50m36.549666043s) [noise=true , address=45.91.116.8:4045 , piece_id=R6YX47QZWG5XANY4WCPDIJC4UBX4WZT2YINYHA4HDIG4YQER5OGQ, seg_pos={0 0}, upload=494352]: Long tail cancellation due to optimal threshold Dec 29 03:19:36.678599880 (563h50m36.549682915s) [seg_pos={0 0}, upload=494352]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 03:19:36.678665305 (563h50m36.54974834s) [upload=494352]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 03:19:36.678671246 (563h50m36.549754281s) [upload=494352]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 03:19:36.965146340 (563h50m36.836229375s) [upload=494352]: Done uploading segments... Dec 29 03:19:36.965164043 (563h50m36.836247088s) [upload=494352]: Upload finished. err: <nil> Duration: 30.004466864s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.001283474s trace: https://jaeger.dev.storj.io/trace/2d4f8cbe7bd1cb2 FAIL (2025-12-29 03:19:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001302631s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)