Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X...............X...................X................] duration: 5.673881086s FAIL (2025-08-13 03:39:44 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 13 03:28:24.870384288 (106h2m19.301464419s) [upload=110366]: Starting upload Aug 13 03:28:24.870426058 (106h2m19.30150618s) [upload=110366]: Done starting upload Aug 13 03:28:24.870488848 (106h2m19.301568979s) [upload=110366]: Uploading segments... Aug 13 03:28:24.870604439 (106h2m19.30168457s) [upload=110366]: Got next segment. Inline: false Aug 13 03:28:24.870637743 (106h2m19.301717874s) [upload=110366]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:28:24.870642031 (106h2m19.301722162s) [upload=110366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 13 03:28:25.321382698 (106h2m19.752462879s) [seg_pos={0 0}, upload=110366]: Begin upload segment... Aug 13 03:28:25.321441521 (106h2m19.752521652s) [seg_pos={0 0}, upload=110366]: Done begin upload segment. Aug 13 03:28:25.321465406 (106h2m19.752545527s) [noise=true , address=104.36.82.44:4876 , piece_id=576P5WMAQ6BCLTG4UAJX2ZCHGAMKCD47Z6YQBWJKHI5MQGGXT4XA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.321492969 (106h2m19.75257309s) [noise=true , address=104.36.82.37:4686 , piece_id=TWWJUP6LY7VX4BW7PSTTZK53JTGSVBP2Y6NQCR3PXT44WCLGY2WA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.321866553 (106h2m19.752946694s) [noise=true , address=104.36.82.4:4833 , piece_id=WOJB675NZW5MONNZ7MHECJEYFRFUQDDQFCG4EATF2DTTQOY5FPIA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322159584 (106h2m19.753239715s) [noise=true , address=104.36.82.38:4716 , piece_id=3Q4BIBCFEYXICWG3CEFEUIBAMBNZWJ7FNCMMB4DHJ73RHKVXGBKQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322826730 (106h2m19.753906862s) [noise=true , address=104.36.82.35:4585 , piece_id=EA7XTERIGFLJMV33KWPJJEFXKDNYHSVJIBYES33AANWX6RFO3Z4A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322949396 (106h2m19.754029517s) [noise=true , address=104.36.82.15:4267 , piece_id=YOBGDLVCGO7Z6XLQQ6K7INK36KDGOHZNA6L2ROJF2VVC3YOE5CEQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323182792 (106h2m19.754262923s) [noise=true , address=64.49.13.139:4473 , piece_id=GYBREOGJD222LYFTSGTYHYLM4T22ZKSQYKN4GNW42ZQSNZQOHQJA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323412270 (106h2m19.754492402s) [noise=true , address=104.36.82.60:4363 , piece_id=SRLXSTGJIEBAM4H6UE6AF32SJY7N2ZHJPPDOXKTUN44ZI3LT2I5A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323740879 (106h2m19.75482101s) [noise=true , address=104.36.82.5:4866 , piece_id=22IXIQNEVDD6VILGPIQMS46JFYVQDI3BNI633KABQLCDBQQB22ZA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.324341969 (106h2m19.755422101s) [noise=true , address=104.36.82.46:4496 , piece_id=K6J6AEVYWM63WM6IFDXSGBUWK6WX72R26RLVLRZ64VSF4H2744JA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.324756232 (106h2m19.755836363s) [noise=true , address=104.36.82.8:4988 , piece_id=KZO4HIEGBYBTXIUMP7YETN5WBBNS2ZMGOA3W6H377RF5RV3RIGRQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325088147 (106h2m19.756168278s) [noise=true , address=64.49.13.139:4478 , piece_id=ZLJZ3QKNWE7LHFZSZEFOSZ6EN27KIAJFEEY7TVL4YW2VZXQM6IOA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325336562 (106h2m19.756416693s) [noise=true , address=104.36.82.36:4641 , piece_id=3A5FI2NFX2Y2QYX6JMEMIPXAYU6ZGB662F6PZGGY6W456MNN4H2Q, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325678096 (106h2m19.756758217s) [noise=true , address=104.36.82.39:4758 , piece_id=6GQJIBOJ6XK6RILWWEPLNQH7OPKSYWFMQG3SXYTVONZRLV3YDU7A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325997056 (106h2m19.757077187s) [noise=true , address=79.127.220.98:4422 , piece_id=R6WFYRTY4IEQ4ORVBDWP5FBP676QRCJDPKSBNJDXL64C2OH2SWDA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326218289 (106h2m19.75729842s) [noise=true , address=104.36.82.21:4488 , piece_id=ZJC77OAGWGPQHJLXGSY2JUS4POTFI4UYW2ORN2OET4U5QCCK2RYA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326380920 (106h2m19.757461051s) [noise=true , address=77.90.31.21:4906 , piece_id=FLIZ63F7NOFHDUEDYOAGMMGEGS473UQPXY2C3WKX67MGWJJEZBIA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326717245 (106h2m19.757797376s) [noise=true , address=104.36.82.58:4298 , piece_id=I76VJFWH36OITPI2LEHTUPWFSC566LKQZBGM7VVJ6744TYN2QESA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:26.509048537 (106h2m20.940128668s) [upload=110366]: Next returned nil segment Aug 13 03:28:26.509066711 (106h2m20.940146842s) [upload=110366]: Waiting for error group managing segments... Aug 13 03:28:26.809943799 (106h2m21.24102393s) [noise=true , address=64.49.13.139:4478 , piece_id=ZLJZ3QKNWE7LHFZSZEFOSZ6EN27KIAJFEEY7TVL4YW2VZXQM6IOA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:26.884546048 (106h2m21.315626179s) [noise=true , address=64.49.13.139:4473 , piece_id=GYBREOGJD222LYFTSGTYHYLM4T22ZKSQYKN4GNW42ZQSNZQOHQJA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.268324420 (106h2m21.699404611s) [noise=true , address=104.36.82.38:4716 , piece_id=3Q4BIBCFEYXICWG3CEFEUIBAMBNZWJ7FNCMMB4DHJ73RHKVXGBKQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.288897151 (106h2m21.719977283s) [noise=true , address=104.36.82.58:4298 , piece_id=I76VJFWH36OITPI2LEHTUPWFSC566LKQZBGM7VVJ6744TYN2QESA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.298496038 (106h2m21.729576169s) [noise=true , address=104.36.82.36:4641 , piece_id=3A5FI2NFX2Y2QYX6JMEMIPXAYU6ZGB662F6PZGGY6W456MNN4H2Q, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.305049777 (106h2m21.736129908s) [noise=true , address=104.36.82.35:4585 , piece_id=EA7XTERIGFLJMV33KWPJJEFXKDNYHSVJIBYES33AANWX6RFO3Z4A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.319397510 (106h2m21.750477631s) [noise=true , address=104.36.82.21:4488 , piece_id=ZJC77OAGWGPQHJLXGSY2JUS4POTFI4UYW2ORN2OET4U5QCCK2RYA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.324171042 (106h2m21.755251173s) [noise=true , address=104.36.82.8:4988 , piece_id=KZO4HIEGBYBTXIUMP7YETN5WBBNS2ZMGOA3W6H377RF5RV3RIGRQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.325044093 (106h2m21.756124224s) [noise=true , address=104.36.82.39:4758 , piece_id=6GQJIBOJ6XK6RILWWEPLNQH7OPKSYWFMQG3SXYTVONZRLV3YDU7A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.326189485 (106h2m21.757269616s) [noise=true , address=104.36.82.4:4833 , piece_id=WOJB675NZW5MONNZ7MHECJEYFRFUQDDQFCG4EATF2DTTQOY5FPIA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.358731589 (106h2m21.78981172s) [noise=true , address=104.36.82.15:4267 , piece_id=YOBGDLVCGO7Z6XLQQ6K7INK36KDGOHZNA6L2ROJF2VVC3YOE5CEQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.364728292 (106h2m21.795808413s) [noise=true , address=104.36.82.37:4686 , piece_id=TWWJUP6LY7VX4BW7PSTTZK53JTGSVBP2Y6NQCR3PXT44WCLGY2WA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.386279758 (106h2m21.817359889s) [noise=true , address=77.90.31.21:4906 , piece_id=FLIZ63F7NOFHDUEDYOAGMMGEGS473UQPXY2C3WKX67MGWJJEZBIA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.393790007 (106h2m21.824870158s) [noise=true , address=104.36.82.44:4876 , piece_id=576P5WMAQ6BCLTG4UAJX2ZCHGAMKCD47Z6YQBWJKHI5MQGGXT4XA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.393863398 (106h2m21.824943529s) [seg_pos={0 0}, upload=110366]: Segment reached optimal threshold of 14 pieces. Aug 13 03:28:27.393993957 (106h2m21.825074088s) [noise=true , address=104.36.82.46:4496 , piece_id=K6J6AEVYWM63WM6IFDXSGBUWK6WX72R26RLVLRZ64VSF4H2744JA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394051778 (106h2m21.82513191s) [noise=true , address=104.36.82.5:4866 , piece_id=22IXIQNEVDD6VILGPIQMS46JFYVQDI3BNI633KABQLCDBQQB22ZA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zUcYfJE8xRW2Tow7bGB3PeNrVBUzVWR82JADkVwPdSr4CTPyr): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394083198 (106h2m21.82516333s) [noise=true , address=104.36.82.60:4363 , piece_id=SRLXSTGJIEBAM4H6UE6AF32SJY7N2ZHJPPDOXKTUN44ZI3LT2I5A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7): protocol: expected piece hash; write tcp 188.166.225.170:42920->104.36.82.60:4363: use of closed network connection; write tcp 188.166.225.170:42920->104.36.82.60:4363: use of closed network connection; EOF Aug 13 03:28:27.394101453 (106h2m21.825181574s) [noise=true , address=79.127.220.98:4422 , piece_id=R6WFYRTY4IEQ4ORVBDWP5FBP676QRCJDPKSBNJDXL64C2OH2SWDA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MScXvmPpGZ6HRRwaoBMrZR3TvhDQh4nZkpSYggRQGzP1JbyVE): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394138204 (106h2m21.825218335s) [seg_pos={0 0}, upload=110366]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 13 03:28:27.394212385 (106h2m21.825292516s) [upload=110366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 13 03:28:27.394217095 (106h2m21.825297226s) [upload=110366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:28:27.720702752 (106h2m22.151782883s) [upload=110366]: Done uploading segments... Aug 13 03:28:27.720720255 (106h2m22.151800386s) [upload=110366]: Upload finished. err: <nil> Duration: 5.67390462s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................X.....XX.........................] duration: 7.546231368s FAIL (2025-08-13 03:39:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.54624713s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 1.828403416s FAIL (2025-08-13 03:39:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 13 03:06:41.898033585 (105h40m36.329113716s) [upload=110007]: Starting upload Aug 13 03:06:41.898093971 (105h40m36.329174102s) [upload=110007]: Done starting upload Aug 13 03:06:41.898140650 (105h40m36.329220781s) [upload=110007]: Uploading segments... Aug 13 03:06:41.898172331 (105h40m36.329252462s) [upload=110007]: Got next segment. Inline: true Aug 13 03:06:41.898176078 (105h40m36.329256209s) [upload=110007]: Waiting for error group managing segments... Aug 13 03:06:41.898208831 (105h40m36.329288962s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:06:41.898212778 (105h40m36.329292909s) [upload=110007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:06:41.898215563 (105h40m36.329295694s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:07:11.244546166 (105h41m5.675626307s) [upload=110007]: Done uploading segments... Aug 13 03:07:11.244585330 (105h41m5.675665482s) [upload=110007]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.828439425s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X...............X...................X...............] duration: 5.900270446s FAIL (2025-08-13 03:39:17 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 13 03:28:24.870384288 (106h2m19.301464419s) [upload=110366]: Starting upload Aug 13 03:28:24.870426058 (106h2m19.30150618s) [upload=110366]: Done starting upload Aug 13 03:28:24.870488848 (106h2m19.301568979s) [upload=110366]: Uploading segments... Aug 13 03:28:24.870604439 (106h2m19.30168457s) [upload=110366]: Got next segment. Inline: false Aug 13 03:28:24.870637743 (106h2m19.301717874s) [upload=110366]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:28:24.870642031 (106h2m19.301722162s) [upload=110366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 13 03:28:25.321382698 (106h2m19.752462879s) [seg_pos={0 0}, upload=110366]: Begin upload segment... Aug 13 03:28:25.321441521 (106h2m19.752521652s) [seg_pos={0 0}, upload=110366]: Done begin upload segment. Aug 13 03:28:25.321465406 (106h2m19.752545527s) [noise=true , address=104.36.82.44:4876 , piece_id=576P5WMAQ6BCLTG4UAJX2ZCHGAMKCD47Z6YQBWJKHI5MQGGXT4XA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.321492969 (106h2m19.75257309s) [noise=true , address=104.36.82.37:4686 , piece_id=TWWJUP6LY7VX4BW7PSTTZK53JTGSVBP2Y6NQCR3PXT44WCLGY2WA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.321866553 (106h2m19.752946694s) [noise=true , address=104.36.82.4:4833 , piece_id=WOJB675NZW5MONNZ7MHECJEYFRFUQDDQFCG4EATF2DTTQOY5FPIA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322159584 (106h2m19.753239715s) [noise=true , address=104.36.82.38:4716 , piece_id=3Q4BIBCFEYXICWG3CEFEUIBAMBNZWJ7FNCMMB4DHJ73RHKVXGBKQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322826730 (106h2m19.753906862s) [noise=true , address=104.36.82.35:4585 , piece_id=EA7XTERIGFLJMV33KWPJJEFXKDNYHSVJIBYES33AANWX6RFO3Z4A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322949396 (106h2m19.754029517s) [noise=true , address=104.36.82.15:4267 , piece_id=YOBGDLVCGO7Z6XLQQ6K7INK36KDGOHZNA6L2ROJF2VVC3YOE5CEQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323182792 (106h2m19.754262923s) [noise=true , address=64.49.13.139:4473 , piece_id=GYBREOGJD222LYFTSGTYHYLM4T22ZKSQYKN4GNW42ZQSNZQOHQJA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323412270 (106h2m19.754492402s) [noise=true , address=104.36.82.60:4363 , piece_id=SRLXSTGJIEBAM4H6UE6AF32SJY7N2ZHJPPDOXKTUN44ZI3LT2I5A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323740879 (106h2m19.75482101s) [noise=true , address=104.36.82.5:4866 , piece_id=22IXIQNEVDD6VILGPIQMS46JFYVQDI3BNI633KABQLCDBQQB22ZA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.324341969 (106h2m19.755422101s) [noise=true , address=104.36.82.46:4496 , piece_id=K6J6AEVYWM63WM6IFDXSGBUWK6WX72R26RLVLRZ64VSF4H2744JA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.324756232 (106h2m19.755836363s) [noise=true , address=104.36.82.8:4988 , piece_id=KZO4HIEGBYBTXIUMP7YETN5WBBNS2ZMGOA3W6H377RF5RV3RIGRQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325088147 (106h2m19.756168278s) [noise=true , address=64.49.13.139:4478 , piece_id=ZLJZ3QKNWE7LHFZSZEFOSZ6EN27KIAJFEEY7TVL4YW2VZXQM6IOA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325336562 (106h2m19.756416693s) [noise=true , address=104.36.82.36:4641 , piece_id=3A5FI2NFX2Y2QYX6JMEMIPXAYU6ZGB662F6PZGGY6W456MNN4H2Q, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325678096 (106h2m19.756758217s) [noise=true , address=104.36.82.39:4758 , piece_id=6GQJIBOJ6XK6RILWWEPLNQH7OPKSYWFMQG3SXYTVONZRLV3YDU7A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325997056 (106h2m19.757077187s) [noise=true , address=79.127.220.98:4422 , piece_id=R6WFYRTY4IEQ4ORVBDWP5FBP676QRCJDPKSBNJDXL64C2OH2SWDA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326218289 (106h2m19.75729842s) [noise=true , address=104.36.82.21:4488 , piece_id=ZJC77OAGWGPQHJLXGSY2JUS4POTFI4UYW2ORN2OET4U5QCCK2RYA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326380920 (106h2m19.757461051s) [noise=true , address=77.90.31.21:4906 , piece_id=FLIZ63F7NOFHDUEDYOAGMMGEGS473UQPXY2C3WKX67MGWJJEZBIA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326717245 (106h2m19.757797376s) [noise=true , address=104.36.82.58:4298 , piece_id=I76VJFWH36OITPI2LEHTUPWFSC566LKQZBGM7VVJ6744TYN2QESA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:26.509048537 (106h2m20.940128668s) [upload=110366]: Next returned nil segment Aug 13 03:28:26.509066711 (106h2m20.940146842s) [upload=110366]: Waiting for error group managing segments... Aug 13 03:28:26.809943799 (106h2m21.24102393s) [noise=true , address=64.49.13.139:4478 , piece_id=ZLJZ3QKNWE7LHFZSZEFOSZ6EN27KIAJFEEY7TVL4YW2VZXQM6IOA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:26.884546048 (106h2m21.315626179s) [noise=true , address=64.49.13.139:4473 , piece_id=GYBREOGJD222LYFTSGTYHYLM4T22ZKSQYKN4GNW42ZQSNZQOHQJA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.268324420 (106h2m21.699404611s) [noise=true , address=104.36.82.38:4716 , piece_id=3Q4BIBCFEYXICWG3CEFEUIBAMBNZWJ7FNCMMB4DHJ73RHKVXGBKQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.288897151 (106h2m21.719977283s) [noise=true , address=104.36.82.58:4298 , piece_id=I76VJFWH36OITPI2LEHTUPWFSC566LKQZBGM7VVJ6744TYN2QESA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.298496038 (106h2m21.729576169s) [noise=true , address=104.36.82.36:4641 , piece_id=3A5FI2NFX2Y2QYX6JMEMIPXAYU6ZGB662F6PZGGY6W456MNN4H2Q, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.305049777 (106h2m21.736129908s) [noise=true , address=104.36.82.35:4585 , piece_id=EA7XTERIGFLJMV33KWPJJEFXKDNYHSVJIBYES33AANWX6RFO3Z4A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.319397510 (106h2m21.750477631s) [noise=true , address=104.36.82.21:4488 , piece_id=ZJC77OAGWGPQHJLXGSY2JUS4POTFI4UYW2ORN2OET4U5QCCK2RYA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.324171042 (106h2m21.755251173s) [noise=true , address=104.36.82.8:4988 , piece_id=KZO4HIEGBYBTXIUMP7YETN5WBBNS2ZMGOA3W6H377RF5RV3RIGRQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.325044093 (106h2m21.756124224s) [noise=true , address=104.36.82.39:4758 , piece_id=6GQJIBOJ6XK6RILWWEPLNQH7OPKSYWFMQG3SXYTVONZRLV3YDU7A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.326189485 (106h2m21.757269616s) [noise=true , address=104.36.82.4:4833 , piece_id=WOJB675NZW5MONNZ7MHECJEYFRFUQDDQFCG4EATF2DTTQOY5FPIA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.358731589 (106h2m21.78981172s) [noise=true , address=104.36.82.15:4267 , piece_id=YOBGDLVCGO7Z6XLQQ6K7INK36KDGOHZNA6L2ROJF2VVC3YOE5CEQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.364728292 (106h2m21.795808413s) [noise=true , address=104.36.82.37:4686 , piece_id=TWWJUP6LY7VX4BW7PSTTZK53JTGSVBP2Y6NQCR3PXT44WCLGY2WA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.386279758 (106h2m21.817359889s) [noise=true , address=77.90.31.21:4906 , piece_id=FLIZ63F7NOFHDUEDYOAGMMGEGS473UQPXY2C3WKX67MGWJJEZBIA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.393790007 (106h2m21.824870158s) [noise=true , address=104.36.82.44:4876 , piece_id=576P5WMAQ6BCLTG4UAJX2ZCHGAMKCD47Z6YQBWJKHI5MQGGXT4XA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.393863398 (106h2m21.824943529s) [seg_pos={0 0}, upload=110366]: Segment reached optimal threshold of 14 pieces. Aug 13 03:28:27.393993957 (106h2m21.825074088s) [noise=true , address=104.36.82.46:4496 , piece_id=K6J6AEVYWM63WM6IFDXSGBUWK6WX72R26RLVLRZ64VSF4H2744JA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394051778 (106h2m21.82513191s) [noise=true , address=104.36.82.5:4866 , piece_id=22IXIQNEVDD6VILGPIQMS46JFYVQDI3BNI633KABQLCDBQQB22ZA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zUcYfJE8xRW2Tow7bGB3PeNrVBUzVWR82JADkVwPdSr4CTPyr): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394083198 (106h2m21.82516333s) [noise=true , address=104.36.82.60:4363 , piece_id=SRLXSTGJIEBAM4H6UE6AF32SJY7N2ZHJPPDOXKTUN44ZI3LT2I5A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7): protocol: expected piece hash; write tcp 188.166.225.170:42920->104.36.82.60:4363: use of closed network connection; write tcp 188.166.225.170:42920->104.36.82.60:4363: use of closed network connection; EOF Aug 13 03:28:27.394101453 (106h2m21.825181574s) [noise=true , address=79.127.220.98:4422 , piece_id=R6WFYRTY4IEQ4ORVBDWP5FBP676QRCJDPKSBNJDXL64C2OH2SWDA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MScXvmPpGZ6HRRwaoBMrZR3TvhDQh4nZkpSYggRQGzP1JbyVE): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394138204 (106h2m21.825218335s) [seg_pos={0 0}, upload=110366]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 13 03:28:27.394212385 (106h2m21.825292516s) [upload=110366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 13 03:28:27.394217095 (106h2m21.825297226s) [upload=110366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:28:27.720702752 (106h2m22.151782883s) [upload=110366]: Done uploading segments... Aug 13 03:28:27.720720255 (106h2m22.151800386s) [upload=110366]: Upload finished. err: <nil> Duration: 5.90029383s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X...............X...................X..............] duration: 5.948722847s FAIL (2025-08-13 03:38:59 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 13 03:28:24.870384288 (106h2m19.301464419s) [upload=110366]: Starting upload Aug 13 03:28:24.870426058 (106h2m19.30150618s) [upload=110366]: Done starting upload Aug 13 03:28:24.870488848 (106h2m19.301568979s) [upload=110366]: Uploading segments... Aug 13 03:28:24.870604439 (106h2m19.30168457s) [upload=110366]: Got next segment. Inline: false Aug 13 03:28:24.870637743 (106h2m19.301717874s) [upload=110366]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:28:24.870642031 (106h2m19.301722162s) [upload=110366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 13 03:28:25.321382698 (106h2m19.752462879s) [seg_pos={0 0}, upload=110366]: Begin upload segment... Aug 13 03:28:25.321441521 (106h2m19.752521652s) [seg_pos={0 0}, upload=110366]: Done begin upload segment. Aug 13 03:28:25.321465406 (106h2m19.752545527s) [noise=true , address=104.36.82.44:4876 , piece_id=576P5WMAQ6BCLTG4UAJX2ZCHGAMKCD47Z6YQBWJKHI5MQGGXT4XA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.321492969 (106h2m19.75257309s) [noise=true , address=104.36.82.37:4686 , piece_id=TWWJUP6LY7VX4BW7PSTTZK53JTGSVBP2Y6NQCR3PXT44WCLGY2WA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.321866553 (106h2m19.752946694s) [noise=true , address=104.36.82.4:4833 , piece_id=WOJB675NZW5MONNZ7MHECJEYFRFUQDDQFCG4EATF2DTTQOY5FPIA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322159584 (106h2m19.753239715s) [noise=true , address=104.36.82.38:4716 , piece_id=3Q4BIBCFEYXICWG3CEFEUIBAMBNZWJ7FNCMMB4DHJ73RHKVXGBKQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322826730 (106h2m19.753906862s) [noise=true , address=104.36.82.35:4585 , piece_id=EA7XTERIGFLJMV33KWPJJEFXKDNYHSVJIBYES33AANWX6RFO3Z4A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322949396 (106h2m19.754029517s) [noise=true , address=104.36.82.15:4267 , piece_id=YOBGDLVCGO7Z6XLQQ6K7INK36KDGOHZNA6L2ROJF2VVC3YOE5CEQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323182792 (106h2m19.754262923s) [noise=true , address=64.49.13.139:4473 , piece_id=GYBREOGJD222LYFTSGTYHYLM4T22ZKSQYKN4GNW42ZQSNZQOHQJA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323412270 (106h2m19.754492402s) [noise=true , address=104.36.82.60:4363 , piece_id=SRLXSTGJIEBAM4H6UE6AF32SJY7N2ZHJPPDOXKTUN44ZI3LT2I5A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323740879 (106h2m19.75482101s) [noise=true , address=104.36.82.5:4866 , piece_id=22IXIQNEVDD6VILGPIQMS46JFYVQDI3BNI633KABQLCDBQQB22ZA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.324341969 (106h2m19.755422101s) [noise=true , address=104.36.82.46:4496 , piece_id=K6J6AEVYWM63WM6IFDXSGBUWK6WX72R26RLVLRZ64VSF4H2744JA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.324756232 (106h2m19.755836363s) [noise=true , address=104.36.82.8:4988 , piece_id=KZO4HIEGBYBTXIUMP7YETN5WBBNS2ZMGOA3W6H377RF5RV3RIGRQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325088147 (106h2m19.756168278s) [noise=true , address=64.49.13.139:4478 , piece_id=ZLJZ3QKNWE7LHFZSZEFOSZ6EN27KIAJFEEY7TVL4YW2VZXQM6IOA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325336562 (106h2m19.756416693s) [noise=true , address=104.36.82.36:4641 , piece_id=3A5FI2NFX2Y2QYX6JMEMIPXAYU6ZGB662F6PZGGY6W456MNN4H2Q, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325678096 (106h2m19.756758217s) [noise=true , address=104.36.82.39:4758 , piece_id=6GQJIBOJ6XK6RILWWEPLNQH7OPKSYWFMQG3SXYTVONZRLV3YDU7A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325997056 (106h2m19.757077187s) [noise=true , address=79.127.220.98:4422 , piece_id=R6WFYRTY4IEQ4ORVBDWP5FBP676QRCJDPKSBNJDXL64C2OH2SWDA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326218289 (106h2m19.75729842s) [noise=true , address=104.36.82.21:4488 , piece_id=ZJC77OAGWGPQHJLXGSY2JUS4POTFI4UYW2ORN2OET4U5QCCK2RYA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326380920 (106h2m19.757461051s) [noise=true , address=77.90.31.21:4906 , piece_id=FLIZ63F7NOFHDUEDYOAGMMGEGS473UQPXY2C3WKX67MGWJJEZBIA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326717245 (106h2m19.757797376s) [noise=true , address=104.36.82.58:4298 , piece_id=I76VJFWH36OITPI2LEHTUPWFSC566LKQZBGM7VVJ6744TYN2QESA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:26.509048537 (106h2m20.940128668s) [upload=110366]: Next returned nil segment Aug 13 03:28:26.509066711 (106h2m20.940146842s) [upload=110366]: Waiting for error group managing segments... Aug 13 03:28:26.809943799 (106h2m21.24102393s) [noise=true , address=64.49.13.139:4478 , piece_id=ZLJZ3QKNWE7LHFZSZEFOSZ6EN27KIAJFEEY7TVL4YW2VZXQM6IOA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:26.884546048 (106h2m21.315626179s) [noise=true , address=64.49.13.139:4473 , piece_id=GYBREOGJD222LYFTSGTYHYLM4T22ZKSQYKN4GNW42ZQSNZQOHQJA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.268324420 (106h2m21.699404611s) [noise=true , address=104.36.82.38:4716 , piece_id=3Q4BIBCFEYXICWG3CEFEUIBAMBNZWJ7FNCMMB4DHJ73RHKVXGBKQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.288897151 (106h2m21.719977283s) [noise=true , address=104.36.82.58:4298 , piece_id=I76VJFWH36OITPI2LEHTUPWFSC566LKQZBGM7VVJ6744TYN2QESA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.298496038 (106h2m21.729576169s) [noise=true , address=104.36.82.36:4641 , piece_id=3A5FI2NFX2Y2QYX6JMEMIPXAYU6ZGB662F6PZGGY6W456MNN4H2Q, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.305049777 (106h2m21.736129908s) [noise=true , address=104.36.82.35:4585 , piece_id=EA7XTERIGFLJMV33KWPJJEFXKDNYHSVJIBYES33AANWX6RFO3Z4A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.319397510 (106h2m21.750477631s) [noise=true , address=104.36.82.21:4488 , piece_id=ZJC77OAGWGPQHJLXGSY2JUS4POTFI4UYW2ORN2OET4U5QCCK2RYA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.324171042 (106h2m21.755251173s) [noise=true , address=104.36.82.8:4988 , piece_id=KZO4HIEGBYBTXIUMP7YETN5WBBNS2ZMGOA3W6H377RF5RV3RIGRQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.325044093 (106h2m21.756124224s) [noise=true , address=104.36.82.39:4758 , piece_id=6GQJIBOJ6XK6RILWWEPLNQH7OPKSYWFMQG3SXYTVONZRLV3YDU7A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.326189485 (106h2m21.757269616s) [noise=true , address=104.36.82.4:4833 , piece_id=WOJB675NZW5MONNZ7MHECJEYFRFUQDDQFCG4EATF2DTTQOY5FPIA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.358731589 (106h2m21.78981172s) [noise=true , address=104.36.82.15:4267 , piece_id=YOBGDLVCGO7Z6XLQQ6K7INK36KDGOHZNA6L2ROJF2VVC3YOE5CEQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.364728292 (106h2m21.795808413s) [noise=true , address=104.36.82.37:4686 , piece_id=TWWJUP6LY7VX4BW7PSTTZK53JTGSVBP2Y6NQCR3PXT44WCLGY2WA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.386279758 (106h2m21.817359889s) [noise=true , address=77.90.31.21:4906 , piece_id=FLIZ63F7NOFHDUEDYOAGMMGEGS473UQPXY2C3WKX67MGWJJEZBIA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.393790007 (106h2m21.824870158s) [noise=true , address=104.36.82.44:4876 , piece_id=576P5WMAQ6BCLTG4UAJX2ZCHGAMKCD47Z6YQBWJKHI5MQGGXT4XA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.393863398 (106h2m21.824943529s) [seg_pos={0 0}, upload=110366]: Segment reached optimal threshold of 14 pieces. Aug 13 03:28:27.393993957 (106h2m21.825074088s) [noise=true , address=104.36.82.46:4496 , piece_id=K6J6AEVYWM63WM6IFDXSGBUWK6WX72R26RLVLRZ64VSF4H2744JA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394051778 (106h2m21.82513191s) [noise=true , address=104.36.82.5:4866 , piece_id=22IXIQNEVDD6VILGPIQMS46JFYVQDI3BNI633KABQLCDBQQB22ZA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zUcYfJE8xRW2Tow7bGB3PeNrVBUzVWR82JADkVwPdSr4CTPyr): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394083198 (106h2m21.82516333s) [noise=true , address=104.36.82.60:4363 , piece_id=SRLXSTGJIEBAM4H6UE6AF32SJY7N2ZHJPPDOXKTUN44ZI3LT2I5A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7): protocol: expected piece hash; write tcp 188.166.225.170:42920->104.36.82.60:4363: use of closed network connection; write tcp 188.166.225.170:42920->104.36.82.60:4363: use of closed network connection; EOF Aug 13 03:28:27.394101453 (106h2m21.825181574s) [noise=true , address=79.127.220.98:4422 , piece_id=R6WFYRTY4IEQ4ORVBDWP5FBP676QRCJDPKSBNJDXL64C2OH2SWDA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MScXvmPpGZ6HRRwaoBMrZR3TvhDQh4nZkpSYggRQGzP1JbyVE): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394138204 (106h2m21.825218335s) [seg_pos={0 0}, upload=110366]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 13 03:28:27.394212385 (106h2m21.825292516s) [upload=110366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 13 03:28:27.394217095 (106h2m21.825297226s) [upload=110366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:28:27.720702752 (106h2m22.151782883s) [upload=110366]: Done uploading segments... Aug 13 03:28:27.720720255 (106h2m22.151800386s) [upload=110366]: Upload finished. err: <nil> Duration: 5.948750871s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 1.908613893s FAIL (2025-08-13 03:38:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 13 03:23:34.987469695 (105h57m29.418549826s) [upload=110284]: Starting upload Aug 13 03:23:34.987520002 (105h57m29.418600133s) [upload=110284]: Done starting upload Aug 13 03:23:34.987570067 (105h57m29.418650198s) [upload=110284]: Uploading segments... Aug 13 03:23:34.987604243 (105h57m29.418684374s) [upload=110284]: Got next segment. Inline: true Aug 13 03:23:34.987608641 (105h57m29.418688772s) [upload=110284]: Waiting for error group managing segments... Aug 13 03:23:34.987640973 (105h57m29.418721094s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:23:34.987644710 (105h57m29.418724841s) [upload=110284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:23:34.987647044 (105h57m29.418727165s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:23:35.199946062 (105h57m29.631026193s) [upload=110284]: Done uploading segments... Aug 13 03:23:35.199964076 (105h57m29.631044207s) [upload=110284]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.908644843s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................X.....XX........................] duration: 7.85450288s FAIL (2025-08-13 03:38:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.854524671s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 1.833544163s FAIL (2025-08-13 03:38:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 13 03:06:41.898033585 (105h40m36.329113716s) [upload=110007]: Starting upload Aug 13 03:06:41.898093971 (105h40m36.329174102s) [upload=110007]: Done starting upload Aug 13 03:06:41.898140650 (105h40m36.329220781s) [upload=110007]: Uploading segments... Aug 13 03:06:41.898172331 (105h40m36.329252462s) [upload=110007]: Got next segment. Inline: true Aug 13 03:06:41.898176078 (105h40m36.329256209s) [upload=110007]: Waiting for error group managing segments... Aug 13 03:06:41.898208831 (105h40m36.329288962s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:06:41.898212778 (105h40m36.329292909s) [upload=110007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:06:41.898215563 (105h40m36.329295694s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:07:11.244546166 (105h41m5.675626307s) [upload=110007]: Done uploading segments... Aug 13 03:07:11.244585330 (105h41m5.675665482s) [upload=110007]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.833565373s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 1.871785719s FAIL (2025-08-13 03:38:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 13 03:23:34.987469695 (105h57m29.418549826s) [upload=110284]: Starting upload Aug 13 03:23:34.987520002 (105h57m29.418600133s) [upload=110284]: Done starting upload Aug 13 03:23:34.987570067 (105h57m29.418650198s) [upload=110284]: Uploading segments... Aug 13 03:23:34.987604243 (105h57m29.418684374s) [upload=110284]: Got next segment. Inline: true Aug 13 03:23:34.987608641 (105h57m29.418688772s) [upload=110284]: Waiting for error group managing segments... Aug 13 03:23:34.987640973 (105h57m29.418721094s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:23:34.987644710 (105h57m29.418724841s) [upload=110284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:23:34.987647044 (105h57m29.418727165s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:23:35.199946062 (105h57m29.631026193s) [upload=110284]: Done uploading segments... Aug 13 03:23:35.199964076 (105h57m29.631044207s) [upload=110284]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.871807852s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X...............X...................X.............] duration: 5.911548393s FAIL (2025-08-13 03:38:16 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 13 03:28:24.870384288 (106h2m19.301464419s) [upload=110366]: Starting upload Aug 13 03:28:24.870426058 (106h2m19.30150618s) [upload=110366]: Done starting upload Aug 13 03:28:24.870488848 (106h2m19.301568979s) [upload=110366]: Uploading segments... Aug 13 03:28:24.870604439 (106h2m19.30168457s) [upload=110366]: Got next segment. Inline: false Aug 13 03:28:24.870637743 (106h2m19.301717874s) [upload=110366]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:28:24.870642031 (106h2m19.301722162s) [upload=110366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 13 03:28:25.321382698 (106h2m19.752462879s) [seg_pos={0 0}, upload=110366]: Begin upload segment... Aug 13 03:28:25.321441521 (106h2m19.752521652s) [seg_pos={0 0}, upload=110366]: Done begin upload segment. Aug 13 03:28:25.321465406 (106h2m19.752545527s) [noise=true , address=104.36.82.44:4876 , piece_id=576P5WMAQ6BCLTG4UAJX2ZCHGAMKCD47Z6YQBWJKHI5MQGGXT4XA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.321492969 (106h2m19.75257309s) [noise=true , address=104.36.82.37:4686 , piece_id=TWWJUP6LY7VX4BW7PSTTZK53JTGSVBP2Y6NQCR3PXT44WCLGY2WA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.321866553 (106h2m19.752946694s) [noise=true , address=104.36.82.4:4833 , piece_id=WOJB675NZW5MONNZ7MHECJEYFRFUQDDQFCG4EATF2DTTQOY5FPIA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322159584 (106h2m19.753239715s) [noise=true , address=104.36.82.38:4716 , piece_id=3Q4BIBCFEYXICWG3CEFEUIBAMBNZWJ7FNCMMB4DHJ73RHKVXGBKQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322826730 (106h2m19.753906862s) [noise=true , address=104.36.82.35:4585 , piece_id=EA7XTERIGFLJMV33KWPJJEFXKDNYHSVJIBYES33AANWX6RFO3Z4A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322949396 (106h2m19.754029517s) [noise=true , address=104.36.82.15:4267 , piece_id=YOBGDLVCGO7Z6XLQQ6K7INK36KDGOHZNA6L2ROJF2VVC3YOE5CEQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323182792 (106h2m19.754262923s) [noise=true , address=64.49.13.139:4473 , piece_id=GYBREOGJD222LYFTSGTYHYLM4T22ZKSQYKN4GNW42ZQSNZQOHQJA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323412270 (106h2m19.754492402s) [noise=true , address=104.36.82.60:4363 , piece_id=SRLXSTGJIEBAM4H6UE6AF32SJY7N2ZHJPPDOXKTUN44ZI3LT2I5A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323740879 (106h2m19.75482101s) [noise=true , address=104.36.82.5:4866 , piece_id=22IXIQNEVDD6VILGPIQMS46JFYVQDI3BNI633KABQLCDBQQB22ZA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.324341969 (106h2m19.755422101s) [noise=true , address=104.36.82.46:4496 , piece_id=K6J6AEVYWM63WM6IFDXSGBUWK6WX72R26RLVLRZ64VSF4H2744JA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.324756232 (106h2m19.755836363s) [noise=true , address=104.36.82.8:4988 , piece_id=KZO4HIEGBYBTXIUMP7YETN5WBBNS2ZMGOA3W6H377RF5RV3RIGRQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325088147 (106h2m19.756168278s) [noise=true , address=64.49.13.139:4478 , piece_id=ZLJZ3QKNWE7LHFZSZEFOSZ6EN27KIAJFEEY7TVL4YW2VZXQM6IOA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325336562 (106h2m19.756416693s) [noise=true , address=104.36.82.36:4641 , piece_id=3A5FI2NFX2Y2QYX6JMEMIPXAYU6ZGB662F6PZGGY6W456MNN4H2Q, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325678096 (106h2m19.756758217s) [noise=true , address=104.36.82.39:4758 , piece_id=6GQJIBOJ6XK6RILWWEPLNQH7OPKSYWFMQG3SXYTVONZRLV3YDU7A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325997056 (106h2m19.757077187s) [noise=true , address=79.127.220.98:4422 , piece_id=R6WFYRTY4IEQ4ORVBDWP5FBP676QRCJDPKSBNJDXL64C2OH2SWDA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326218289 (106h2m19.75729842s) [noise=true , address=104.36.82.21:4488 , piece_id=ZJC77OAGWGPQHJLXGSY2JUS4POTFI4UYW2ORN2OET4U5QCCK2RYA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326380920 (106h2m19.757461051s) [noise=true , address=77.90.31.21:4906 , piece_id=FLIZ63F7NOFHDUEDYOAGMMGEGS473UQPXY2C3WKX67MGWJJEZBIA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326717245 (106h2m19.757797376s) [noise=true , address=104.36.82.58:4298 , piece_id=I76VJFWH36OITPI2LEHTUPWFSC566LKQZBGM7VVJ6744TYN2QESA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:26.509048537 (106h2m20.940128668s) [upload=110366]: Next returned nil segment Aug 13 03:28:26.509066711 (106h2m20.940146842s) [upload=110366]: Waiting for error group managing segments... Aug 13 03:28:26.809943799 (106h2m21.24102393s) [noise=true , address=64.49.13.139:4478 , piece_id=ZLJZ3QKNWE7LHFZSZEFOSZ6EN27KIAJFEEY7TVL4YW2VZXQM6IOA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:26.884546048 (106h2m21.315626179s) [noise=true , address=64.49.13.139:4473 , piece_id=GYBREOGJD222LYFTSGTYHYLM4T22ZKSQYKN4GNW42ZQSNZQOHQJA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.268324420 (106h2m21.699404611s) [noise=true , address=104.36.82.38:4716 , piece_id=3Q4BIBCFEYXICWG3CEFEUIBAMBNZWJ7FNCMMB4DHJ73RHKVXGBKQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.288897151 (106h2m21.719977283s) [noise=true , address=104.36.82.58:4298 , piece_id=I76VJFWH36OITPI2LEHTUPWFSC566LKQZBGM7VVJ6744TYN2QESA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.298496038 (106h2m21.729576169s) [noise=true , address=104.36.82.36:4641 , piece_id=3A5FI2NFX2Y2QYX6JMEMIPXAYU6ZGB662F6PZGGY6W456MNN4H2Q, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.305049777 (106h2m21.736129908s) [noise=true , address=104.36.82.35:4585 , piece_id=EA7XTERIGFLJMV33KWPJJEFXKDNYHSVJIBYES33AANWX6RFO3Z4A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.319397510 (106h2m21.750477631s) [noise=true , address=104.36.82.21:4488 , piece_id=ZJC77OAGWGPQHJLXGSY2JUS4POTFI4UYW2ORN2OET4U5QCCK2RYA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.324171042 (106h2m21.755251173s) [noise=true , address=104.36.82.8:4988 , piece_id=KZO4HIEGBYBTXIUMP7YETN5WBBNS2ZMGOA3W6H377RF5RV3RIGRQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.325044093 (106h2m21.756124224s) [noise=true , address=104.36.82.39:4758 , piece_id=6GQJIBOJ6XK6RILWWEPLNQH7OPKSYWFMQG3SXYTVONZRLV3YDU7A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.326189485 (106h2m21.757269616s) [noise=true , address=104.36.82.4:4833 , piece_id=WOJB675NZW5MONNZ7MHECJEYFRFUQDDQFCG4EATF2DTTQOY5FPIA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.358731589 (106h2m21.78981172s) [noise=true , address=104.36.82.15:4267 , piece_id=YOBGDLVCGO7Z6XLQQ6K7INK36KDGOHZNA6L2ROJF2VVC3YOE5CEQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.364728292 (106h2m21.795808413s) [noise=true , address=104.36.82.37:4686 , piece_id=TWWJUP6LY7VX4BW7PSTTZK53JTGSVBP2Y6NQCR3PXT44WCLGY2WA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.386279758 (106h2m21.817359889s) [noise=true , address=77.90.31.21:4906 , piece_id=FLIZ63F7NOFHDUEDYOAGMMGEGS473UQPXY2C3WKX67MGWJJEZBIA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.393790007 (106h2m21.824870158s) [noise=true , address=104.36.82.44:4876 , piece_id=576P5WMAQ6BCLTG4UAJX2ZCHGAMKCD47Z6YQBWJKHI5MQGGXT4XA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.393863398 (106h2m21.824943529s) [seg_pos={0 0}, upload=110366]: Segment reached optimal threshold of 14 pieces. Aug 13 03:28:27.393993957 (106h2m21.825074088s) [noise=true , address=104.36.82.46:4496 , piece_id=K6J6AEVYWM63WM6IFDXSGBUWK6WX72R26RLVLRZ64VSF4H2744JA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394051778 (106h2m21.82513191s) [noise=true , address=104.36.82.5:4866 , piece_id=22IXIQNEVDD6VILGPIQMS46JFYVQDI3BNI633KABQLCDBQQB22ZA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zUcYfJE8xRW2Tow7bGB3PeNrVBUzVWR82JADkVwPdSr4CTPyr): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394083198 (106h2m21.82516333s) [noise=true , address=104.36.82.60:4363 , piece_id=SRLXSTGJIEBAM4H6UE6AF32SJY7N2ZHJPPDOXKTUN44ZI3LT2I5A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7): protocol: expected piece hash; write tcp 188.166.225.170:42920->104.36.82.60:4363: use of closed network connection; write tcp 188.166.225.170:42920->104.36.82.60:4363: use of closed network connection; EOF Aug 13 03:28:27.394101453 (106h2m21.825181574s) [noise=true , address=79.127.220.98:4422 , piece_id=R6WFYRTY4IEQ4ORVBDWP5FBP676QRCJDPKSBNJDXL64C2OH2SWDA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MScXvmPpGZ6HRRwaoBMrZR3TvhDQh4nZkpSYggRQGzP1JbyVE): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394138204 (106h2m21.825218335s) [seg_pos={0 0}, upload=110366]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 13 03:28:27.394212385 (106h2m21.825292516s) [upload=110366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 13 03:28:27.394217095 (106h2m21.825297226s) [upload=110366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:28:27.720702752 (106h2m22.151782883s) [upload=110366]: Done uploading segments... Aug 13 03:28:27.720720255 (106h2m22.151800386s) [upload=110366]: Upload finished. err: <nil> Duration: 5.911597767s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 1.94008031s FAIL (2025-08-13 03:38:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 13 03:06:41.898033585 (105h40m36.329113716s) [upload=110007]: Starting upload Aug 13 03:06:41.898093971 (105h40m36.329174102s) [upload=110007]: Done starting upload Aug 13 03:06:41.898140650 (105h40m36.329220781s) [upload=110007]: Uploading segments... Aug 13 03:06:41.898172331 (105h40m36.329252462s) [upload=110007]: Got next segment. Inline: true Aug 13 03:06:41.898176078 (105h40m36.329256209s) [upload=110007]: Waiting for error group managing segments... Aug 13 03:06:41.898208831 (105h40m36.329288962s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:06:41.898212778 (105h40m36.329292909s) [upload=110007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:06:41.898215563 (105h40m36.329295694s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:07:11.244546166 (105h41m5.675626307s) [upload=110007]: Done uploading segments... Aug 13 03:07:11.244585330 (105h41m5.675665482s) [upload=110007]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.940125697s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 1.874351609s FAIL (2025-08-13 03:37:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 13 03:23:34.987469695 (105h57m29.418549826s) [upload=110284]: Starting upload Aug 13 03:23:34.987520002 (105h57m29.418600133s) [upload=110284]: Done starting upload Aug 13 03:23:34.987570067 (105h57m29.418650198s) [upload=110284]: Uploading segments... Aug 13 03:23:34.987604243 (105h57m29.418684374s) [upload=110284]: Got next segment. Inline: true Aug 13 03:23:34.987608641 (105h57m29.418688772s) [upload=110284]: Waiting for error group managing segments... Aug 13 03:23:34.987640973 (105h57m29.418721094s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:23:34.987644710 (105h57m29.418724841s) [upload=110284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:23:34.987647044 (105h57m29.418727165s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:23:35.199946062 (105h57m29.631026193s) [upload=110284]: Done uploading segments... Aug 13 03:23:35.199964076 (105h57m29.631044207s) [upload=110284]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.8743782s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................X.....XX.......................] duration: 10.291800402s FAIL (2025-08-13 03:37:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.291855437s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 1.904004417s FAIL (2025-08-13 03:37:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 13 03:06:41.898033585 (105h40m36.329113716s) [upload=110007]: Starting upload Aug 13 03:06:41.898093971 (105h40m36.329174102s) [upload=110007]: Done starting upload Aug 13 03:06:41.898140650 (105h40m36.329220781s) [upload=110007]: Uploading segments... Aug 13 03:06:41.898172331 (105h40m36.329252462s) [upload=110007]: Got next segment. Inline: true Aug 13 03:06:41.898176078 (105h40m36.329256209s) [upload=110007]: Waiting for error group managing segments... Aug 13 03:06:41.898208831 (105h40m36.329288962s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:06:41.898212778 (105h40m36.329292909s) [upload=110007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:06:41.898215563 (105h40m36.329295694s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:07:11.244546166 (105h41m5.675626307s) [upload=110007]: Done uploading segments... Aug 13 03:07:11.244585330 (105h41m5.675665482s) [upload=110007]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.904027452s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X...............X...................X............] duration: 5.86633575s FAIL (2025-08-13 03:37:34 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 13 03:28:24.870384288 (106h2m19.301464419s) [upload=110366]: Starting upload Aug 13 03:28:24.870426058 (106h2m19.30150618s) [upload=110366]: Done starting upload Aug 13 03:28:24.870488848 (106h2m19.301568979s) [upload=110366]: Uploading segments... Aug 13 03:28:24.870604439 (106h2m19.30168457s) [upload=110366]: Got next segment. Inline: false Aug 13 03:28:24.870637743 (106h2m19.301717874s) [upload=110366]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:28:24.870642031 (106h2m19.301722162s) [upload=110366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 13 03:28:25.321382698 (106h2m19.752462879s) [seg_pos={0 0}, upload=110366]: Begin upload segment... Aug 13 03:28:25.321441521 (106h2m19.752521652s) [seg_pos={0 0}, upload=110366]: Done begin upload segment. Aug 13 03:28:25.321465406 (106h2m19.752545527s) [noise=true , address=104.36.82.44:4876 , piece_id=576P5WMAQ6BCLTG4UAJX2ZCHGAMKCD47Z6YQBWJKHI5MQGGXT4XA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.321492969 (106h2m19.75257309s) [noise=true , address=104.36.82.37:4686 , piece_id=TWWJUP6LY7VX4BW7PSTTZK53JTGSVBP2Y6NQCR3PXT44WCLGY2WA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.321866553 (106h2m19.752946694s) [noise=true , address=104.36.82.4:4833 , piece_id=WOJB675NZW5MONNZ7MHECJEYFRFUQDDQFCG4EATF2DTTQOY5FPIA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322159584 (106h2m19.753239715s) [noise=true , address=104.36.82.38:4716 , piece_id=3Q4BIBCFEYXICWG3CEFEUIBAMBNZWJ7FNCMMB4DHJ73RHKVXGBKQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322826730 (106h2m19.753906862s) [noise=true , address=104.36.82.35:4585 , piece_id=EA7XTERIGFLJMV33KWPJJEFXKDNYHSVJIBYES33AANWX6RFO3Z4A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322949396 (106h2m19.754029517s) [noise=true , address=104.36.82.15:4267 , piece_id=YOBGDLVCGO7Z6XLQQ6K7INK36KDGOHZNA6L2ROJF2VVC3YOE5CEQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323182792 (106h2m19.754262923s) [noise=true , address=64.49.13.139:4473 , piece_id=GYBREOGJD222LYFTSGTYHYLM4T22ZKSQYKN4GNW42ZQSNZQOHQJA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323412270 (106h2m19.754492402s) [noise=true , address=104.36.82.60:4363 , piece_id=SRLXSTGJIEBAM4H6UE6AF32SJY7N2ZHJPPDOXKTUN44ZI3LT2I5A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323740879 (106h2m19.75482101s) [noise=true , address=104.36.82.5:4866 , piece_id=22IXIQNEVDD6VILGPIQMS46JFYVQDI3BNI633KABQLCDBQQB22ZA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.324341969 (106h2m19.755422101s) [noise=true , address=104.36.82.46:4496 , piece_id=K6J6AEVYWM63WM6IFDXSGBUWK6WX72R26RLVLRZ64VSF4H2744JA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.324756232 (106h2m19.755836363s) [noise=true , address=104.36.82.8:4988 , piece_id=KZO4HIEGBYBTXIUMP7YETN5WBBNS2ZMGOA3W6H377RF5RV3RIGRQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325088147 (106h2m19.756168278s) [noise=true , address=64.49.13.139:4478 , piece_id=ZLJZ3QKNWE7LHFZSZEFOSZ6EN27KIAJFEEY7TVL4YW2VZXQM6IOA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325336562 (106h2m19.756416693s) [noise=true , address=104.36.82.36:4641 , piece_id=3A5FI2NFX2Y2QYX6JMEMIPXAYU6ZGB662F6PZGGY6W456MNN4H2Q, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325678096 (106h2m19.756758217s) [noise=true , address=104.36.82.39:4758 , piece_id=6GQJIBOJ6XK6RILWWEPLNQH7OPKSYWFMQG3SXYTVONZRLV3YDU7A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325997056 (106h2m19.757077187s) [noise=true , address=79.127.220.98:4422 , piece_id=R6WFYRTY4IEQ4ORVBDWP5FBP676QRCJDPKSBNJDXL64C2OH2SWDA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326218289 (106h2m19.75729842s) [noise=true , address=104.36.82.21:4488 , piece_id=ZJC77OAGWGPQHJLXGSY2JUS4POTFI4UYW2ORN2OET4U5QCCK2RYA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326380920 (106h2m19.757461051s) [noise=true , address=77.90.31.21:4906 , piece_id=FLIZ63F7NOFHDUEDYOAGMMGEGS473UQPXY2C3WKX67MGWJJEZBIA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326717245 (106h2m19.757797376s) [noise=true , address=104.36.82.58:4298 , piece_id=I76VJFWH36OITPI2LEHTUPWFSC566LKQZBGM7VVJ6744TYN2QESA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:26.509048537 (106h2m20.940128668s) [upload=110366]: Next returned nil segment Aug 13 03:28:26.509066711 (106h2m20.940146842s) [upload=110366]: Waiting for error group managing segments... Aug 13 03:28:26.809943799 (106h2m21.24102393s) [noise=true , address=64.49.13.139:4478 , piece_id=ZLJZ3QKNWE7LHFZSZEFOSZ6EN27KIAJFEEY7TVL4YW2VZXQM6IOA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:26.884546048 (106h2m21.315626179s) [noise=true , address=64.49.13.139:4473 , piece_id=GYBREOGJD222LYFTSGTYHYLM4T22ZKSQYKN4GNW42ZQSNZQOHQJA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.268324420 (106h2m21.699404611s) [noise=true , address=104.36.82.38:4716 , piece_id=3Q4BIBCFEYXICWG3CEFEUIBAMBNZWJ7FNCMMB4DHJ73RHKVXGBKQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.288897151 (106h2m21.719977283s) [noise=true , address=104.36.82.58:4298 , piece_id=I76VJFWH36OITPI2LEHTUPWFSC566LKQZBGM7VVJ6744TYN2QESA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.298496038 (106h2m21.729576169s) [noise=true , address=104.36.82.36:4641 , piece_id=3A5FI2NFX2Y2QYX6JMEMIPXAYU6ZGB662F6PZGGY6W456MNN4H2Q, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.305049777 (106h2m21.736129908s) [noise=true , address=104.36.82.35:4585 , piece_id=EA7XTERIGFLJMV33KWPJJEFXKDNYHSVJIBYES33AANWX6RFO3Z4A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.319397510 (106h2m21.750477631s) [noise=true , address=104.36.82.21:4488 , piece_id=ZJC77OAGWGPQHJLXGSY2JUS4POTFI4UYW2ORN2OET4U5QCCK2RYA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.324171042 (106h2m21.755251173s) [noise=true , address=104.36.82.8:4988 , piece_id=KZO4HIEGBYBTXIUMP7YETN5WBBNS2ZMGOA3W6H377RF5RV3RIGRQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.325044093 (106h2m21.756124224s) [noise=true , address=104.36.82.39:4758 , piece_id=6GQJIBOJ6XK6RILWWEPLNQH7OPKSYWFMQG3SXYTVONZRLV3YDU7A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.326189485 (106h2m21.757269616s) [noise=true , address=104.36.82.4:4833 , piece_id=WOJB675NZW5MONNZ7MHECJEYFRFUQDDQFCG4EATF2DTTQOY5FPIA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.358731589 (106h2m21.78981172s) [noise=true , address=104.36.82.15:4267 , piece_id=YOBGDLVCGO7Z6XLQQ6K7INK36KDGOHZNA6L2ROJF2VVC3YOE5CEQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.364728292 (106h2m21.795808413s) [noise=true , address=104.36.82.37:4686 , piece_id=TWWJUP6LY7VX4BW7PSTTZK53JTGSVBP2Y6NQCR3PXT44WCLGY2WA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.386279758 (106h2m21.817359889s) [noise=true , address=77.90.31.21:4906 , piece_id=FLIZ63F7NOFHDUEDYOAGMMGEGS473UQPXY2C3WKX67MGWJJEZBIA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.393790007 (106h2m21.824870158s) [noise=true , address=104.36.82.44:4876 , piece_id=576P5WMAQ6BCLTG4UAJX2ZCHGAMKCD47Z6YQBWJKHI5MQGGXT4XA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.393863398 (106h2m21.824943529s) [seg_pos={0 0}, upload=110366]: Segment reached optimal threshold of 14 pieces. Aug 13 03:28:27.393993957 (106h2m21.825074088s) [noise=true , address=104.36.82.46:4496 , piece_id=K6J6AEVYWM63WM6IFDXSGBUWK6WX72R26RLVLRZ64VSF4H2744JA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394051778 (106h2m21.82513191s) [noise=true , address=104.36.82.5:4866 , piece_id=22IXIQNEVDD6VILGPIQMS46JFYVQDI3BNI633KABQLCDBQQB22ZA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zUcYfJE8xRW2Tow7bGB3PeNrVBUzVWR82JADkVwPdSr4CTPyr): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394083198 (106h2m21.82516333s) [noise=true , address=104.36.82.60:4363 , piece_id=SRLXSTGJIEBAM4H6UE6AF32SJY7N2ZHJPPDOXKTUN44ZI3LT2I5A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7): protocol: expected piece hash; write tcp 188.166.225.170:42920->104.36.82.60:4363: use of closed network connection; write tcp 188.166.225.170:42920->104.36.82.60:4363: use of closed network connection; EOF Aug 13 03:28:27.394101453 (106h2m21.825181574s) [noise=true , address=79.127.220.98:4422 , piece_id=R6WFYRTY4IEQ4ORVBDWP5FBP676QRCJDPKSBNJDXL64C2OH2SWDA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MScXvmPpGZ6HRRwaoBMrZR3TvhDQh4nZkpSYggRQGzP1JbyVE): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394138204 (106h2m21.825218335s) [seg_pos={0 0}, upload=110366]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 13 03:28:27.394212385 (106h2m21.825292516s) [upload=110366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 13 03:28:27.394217095 (106h2m21.825297226s) [upload=110366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:28:27.720702752 (106h2m22.151782883s) [upload=110366]: Done uploading segments... Aug 13 03:28:27.720720255 (106h2m22.151800386s) [upload=110366]: Upload finished. err: <nil> Duration: 5.866365175s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 1.83648533s FAIL (2025-08-13 03:37:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 13 03:23:34.987469695 (105h57m29.418549826s) [upload=110284]: Starting upload Aug 13 03:23:34.987520002 (105h57m29.418600133s) [upload=110284]: Done starting upload Aug 13 03:23:34.987570067 (105h57m29.418650198s) [upload=110284]: Uploading segments... Aug 13 03:23:34.987604243 (105h57m29.418684374s) [upload=110284]: Got next segment. Inline: true Aug 13 03:23:34.987608641 (105h57m29.418688772s) [upload=110284]: Waiting for error group managing segments... Aug 13 03:23:34.987640973 (105h57m29.418721094s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:23:34.987644710 (105h57m29.418724841s) [upload=110284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:23:34.987647044 (105h57m29.418727165s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:23:35.199946062 (105h57m29.631026193s) [upload=110284]: Done uploading segments... Aug 13 03:23:35.199964076 (105h57m29.631044207s) [upload=110284]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.836512845s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................X.....XX......................] duration: 7.758163808s FAIL (2025-08-13 03:37:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.758184176s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [X...........X...............X...................X...........] duration: 5.937023021s FAIL (2025-08-13 03:37:14 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 13 03:28:24.870384288 (106h2m19.301464419s) [upload=110366]: Starting upload Aug 13 03:28:24.870426058 (106h2m19.30150618s) [upload=110366]: Done starting upload Aug 13 03:28:24.870488848 (106h2m19.301568979s) [upload=110366]: Uploading segments... Aug 13 03:28:24.870604439 (106h2m19.30168457s) [upload=110366]: Got next segment. Inline: false Aug 13 03:28:24.870637743 (106h2m19.301717874s) [upload=110366]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:28:24.870642031 (106h2m19.301722162s) [upload=110366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 13 03:28:25.321382698 (106h2m19.752462879s) [seg_pos={0 0}, upload=110366]: Begin upload segment... Aug 13 03:28:25.321441521 (106h2m19.752521652s) [seg_pos={0 0}, upload=110366]: Done begin upload segment. Aug 13 03:28:25.321465406 (106h2m19.752545527s) [noise=true , address=104.36.82.44:4876 , piece_id=576P5WMAQ6BCLTG4UAJX2ZCHGAMKCD47Z6YQBWJKHI5MQGGXT4XA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.321492969 (106h2m19.75257309s) [noise=true , address=104.36.82.37:4686 , piece_id=TWWJUP6LY7VX4BW7PSTTZK53JTGSVBP2Y6NQCR3PXT44WCLGY2WA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.321866553 (106h2m19.752946694s) [noise=true , address=104.36.82.4:4833 , piece_id=WOJB675NZW5MONNZ7MHECJEYFRFUQDDQFCG4EATF2DTTQOY5FPIA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322159584 (106h2m19.753239715s) [noise=true , address=104.36.82.38:4716 , piece_id=3Q4BIBCFEYXICWG3CEFEUIBAMBNZWJ7FNCMMB4DHJ73RHKVXGBKQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322826730 (106h2m19.753906862s) [noise=true , address=104.36.82.35:4585 , piece_id=EA7XTERIGFLJMV33KWPJJEFXKDNYHSVJIBYES33AANWX6RFO3Z4A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322949396 (106h2m19.754029517s) [noise=true , address=104.36.82.15:4267 , piece_id=YOBGDLVCGO7Z6XLQQ6K7INK36KDGOHZNA6L2ROJF2VVC3YOE5CEQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323182792 (106h2m19.754262923s) [noise=true , address=64.49.13.139:4473 , piece_id=GYBREOGJD222LYFTSGTYHYLM4T22ZKSQYKN4GNW42ZQSNZQOHQJA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323412270 (106h2m19.754492402s) [noise=true , address=104.36.82.60:4363 , piece_id=SRLXSTGJIEBAM4H6UE6AF32SJY7N2ZHJPPDOXKTUN44ZI3LT2I5A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323740879 (106h2m19.75482101s) [noise=true , address=104.36.82.5:4866 , piece_id=22IXIQNEVDD6VILGPIQMS46JFYVQDI3BNI633KABQLCDBQQB22ZA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.324341969 (106h2m19.755422101s) [noise=true , address=104.36.82.46:4496 , piece_id=K6J6AEVYWM63WM6IFDXSGBUWK6WX72R26RLVLRZ64VSF4H2744JA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.324756232 (106h2m19.755836363s) [noise=true , address=104.36.82.8:4988 , piece_id=KZO4HIEGBYBTXIUMP7YETN5WBBNS2ZMGOA3W6H377RF5RV3RIGRQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325088147 (106h2m19.756168278s) [noise=true , address=64.49.13.139:4478 , piece_id=ZLJZ3QKNWE7LHFZSZEFOSZ6EN27KIAJFEEY7TVL4YW2VZXQM6IOA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325336562 (106h2m19.756416693s) [noise=true , address=104.36.82.36:4641 , piece_id=3A5FI2NFX2Y2QYX6JMEMIPXAYU6ZGB662F6PZGGY6W456MNN4H2Q, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325678096 (106h2m19.756758217s) [noise=true , address=104.36.82.39:4758 , piece_id=6GQJIBOJ6XK6RILWWEPLNQH7OPKSYWFMQG3SXYTVONZRLV3YDU7A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325997056 (106h2m19.757077187s) [noise=true , address=79.127.220.98:4422 , piece_id=R6WFYRTY4IEQ4ORVBDWP5FBP676QRCJDPKSBNJDXL64C2OH2SWDA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326218289 (106h2m19.75729842s) [noise=true , address=104.36.82.21:4488 , piece_id=ZJC77OAGWGPQHJLXGSY2JUS4POTFI4UYW2ORN2OET4U5QCCK2RYA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326380920 (106h2m19.757461051s) [noise=true , address=77.90.31.21:4906 , piece_id=FLIZ63F7NOFHDUEDYOAGMMGEGS473UQPXY2C3WKX67MGWJJEZBIA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326717245 (106h2m19.757797376s) [noise=true , address=104.36.82.58:4298 , piece_id=I76VJFWH36OITPI2LEHTUPWFSC566LKQZBGM7VVJ6744TYN2QESA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:26.509048537 (106h2m20.940128668s) [upload=110366]: Next returned nil segment Aug 13 03:28:26.509066711 (106h2m20.940146842s) [upload=110366]: Waiting for error group managing segments... Aug 13 03:28:26.809943799 (106h2m21.24102393s) [noise=true , address=64.49.13.139:4478 , piece_id=ZLJZ3QKNWE7LHFZSZEFOSZ6EN27KIAJFEEY7TVL4YW2VZXQM6IOA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:26.884546048 (106h2m21.315626179s) [noise=true , address=64.49.13.139:4473 , piece_id=GYBREOGJD222LYFTSGTYHYLM4T22ZKSQYKN4GNW42ZQSNZQOHQJA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.268324420 (106h2m21.699404611s) [noise=true , address=104.36.82.38:4716 , piece_id=3Q4BIBCFEYXICWG3CEFEUIBAMBNZWJ7FNCMMB4DHJ73RHKVXGBKQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.288897151 (106h2m21.719977283s) [noise=true , address=104.36.82.58:4298 , piece_id=I76VJFWH36OITPI2LEHTUPWFSC566LKQZBGM7VVJ6744TYN2QESA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.298496038 (106h2m21.729576169s) [noise=true , address=104.36.82.36:4641 , piece_id=3A5FI2NFX2Y2QYX6JMEMIPXAYU6ZGB662F6PZGGY6W456MNN4H2Q, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.305049777 (106h2m21.736129908s) [noise=true , address=104.36.82.35:4585 , piece_id=EA7XTERIGFLJMV33KWPJJEFXKDNYHSVJIBYES33AANWX6RFO3Z4A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.319397510 (106h2m21.750477631s) [noise=true , address=104.36.82.21:4488 , piece_id=ZJC77OAGWGPQHJLXGSY2JUS4POTFI4UYW2ORN2OET4U5QCCK2RYA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.324171042 (106h2m21.755251173s) [noise=true , address=104.36.82.8:4988 , piece_id=KZO4HIEGBYBTXIUMP7YETN5WBBNS2ZMGOA3W6H377RF5RV3RIGRQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.325044093 (106h2m21.756124224s) [noise=true , address=104.36.82.39:4758 , piece_id=6GQJIBOJ6XK6RILWWEPLNQH7OPKSYWFMQG3SXYTVONZRLV3YDU7A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.326189485 (106h2m21.757269616s) [noise=true , address=104.36.82.4:4833 , piece_id=WOJB675NZW5MONNZ7MHECJEYFRFUQDDQFCG4EATF2DTTQOY5FPIA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.358731589 (106h2m21.78981172s) [noise=true , address=104.36.82.15:4267 , piece_id=YOBGDLVCGO7Z6XLQQ6K7INK36KDGOHZNA6L2ROJF2VVC3YOE5CEQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.364728292 (106h2m21.795808413s) [noise=true , address=104.36.82.37:4686 , piece_id=TWWJUP6LY7VX4BW7PSTTZK53JTGSVBP2Y6NQCR3PXT44WCLGY2WA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.386279758 (106h2m21.817359889s) [noise=true , address=77.90.31.21:4906 , piece_id=FLIZ63F7NOFHDUEDYOAGMMGEGS473UQPXY2C3WKX67MGWJJEZBIA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.393790007 (106h2m21.824870158s) [noise=true , address=104.36.82.44:4876 , piece_id=576P5WMAQ6BCLTG4UAJX2ZCHGAMKCD47Z6YQBWJKHI5MQGGXT4XA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.393863398 (106h2m21.824943529s) [seg_pos={0 0}, upload=110366]: Segment reached optimal threshold of 14 pieces. Aug 13 03:28:27.393993957 (106h2m21.825074088s) [noise=true , address=104.36.82.46:4496 , piece_id=K6J6AEVYWM63WM6IFDXSGBUWK6WX72R26RLVLRZ64VSF4H2744JA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394051778 (106h2m21.82513191s) [noise=true , address=104.36.82.5:4866 , piece_id=22IXIQNEVDD6VILGPIQMS46JFYVQDI3BNI633KABQLCDBQQB22ZA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zUcYfJE8xRW2Tow7bGB3PeNrVBUzVWR82JADkVwPdSr4CTPyr): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394083198 (106h2m21.82516333s) [noise=true , address=104.36.82.60:4363 , piece_id=SRLXSTGJIEBAM4H6UE6AF32SJY7N2ZHJPPDOXKTUN44ZI3LT2I5A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7): protocol: expected piece hash; write tcp 188.166.225.170:42920->104.36.82.60:4363: use of closed network connection; write tcp 188.166.225.170:42920->104.36.82.60:4363: use of closed network connection; EOF Aug 13 03:28:27.394101453 (106h2m21.825181574s) [noise=true , address=79.127.220.98:4422 , piece_id=R6WFYRTY4IEQ4ORVBDWP5FBP676QRCJDPKSBNJDXL64C2OH2SWDA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MScXvmPpGZ6HRRwaoBMrZR3TvhDQh4nZkpSYggRQGzP1JbyVE): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394138204 (106h2m21.825218335s) [seg_pos={0 0}, upload=110366]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 13 03:28:27.394212385 (106h2m21.825292516s) [upload=110366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 13 03:28:27.394217095 (106h2m21.825297226s) [upload=110366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:28:27.720702752 (106h2m22.151782883s) [upload=110366]: Done uploading segments... Aug 13 03:28:27.720720255 (106h2m22.151800386s) [upload=110366]: Upload finished. err: <nil> Duration: 5.937052096s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 1.925741676s FAIL (2025-08-13 03:36:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 13 03:23:34.987469695 (105h57m29.418549826s) [upload=110284]: Starting upload Aug 13 03:23:34.987520002 (105h57m29.418600133s) [upload=110284]: Done starting upload Aug 13 03:23:34.987570067 (105h57m29.418650198s) [upload=110284]: Uploading segments... Aug 13 03:23:34.987604243 (105h57m29.418684374s) [upload=110284]: Got next segment. Inline: true Aug 13 03:23:34.987608641 (105h57m29.418688772s) [upload=110284]: Waiting for error group managing segments... Aug 13 03:23:34.987640973 (105h57m29.418721094s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:23:34.987644710 (105h57m29.418724841s) [upload=110284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:23:34.987647044 (105h57m29.418727165s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:23:35.199946062 (105h57m29.631026193s) [upload=110284]: Done uploading segments... Aug 13 03:23:35.199964076 (105h57m29.631044207s) [upload=110284]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.925772453s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................X.....XX.....................] duration: 7.294458942s FAIL (2025-08-13 03:36:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.29448414s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 1.923634404s FAIL (2025-08-13 03:36:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 13 03:06:41.898033585 (105h40m36.329113716s) [upload=110007]: Starting upload Aug 13 03:06:41.898093971 (105h40m36.329174102s) [upload=110007]: Done starting upload Aug 13 03:06:41.898140650 (105h40m36.329220781s) [upload=110007]: Uploading segments... Aug 13 03:06:41.898172331 (105h40m36.329252462s) [upload=110007]: Got next segment. Inline: true Aug 13 03:06:41.898176078 (105h40m36.329256209s) [upload=110007]: Waiting for error group managing segments... Aug 13 03:06:41.898208831 (105h40m36.329288962s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:06:41.898212778 (105h40m36.329292909s) [upload=110007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:06:41.898215563 (105h40m36.329295694s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:07:11.244546166 (105h41m5.675626307s) [upload=110007]: Done uploading segments... Aug 13 03:07:11.244585330 (105h41m5.675665482s) [upload=110007]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.92365963s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X...........X...............X...................X..........] duration: 5.600746293s FAIL (2025-08-13 03:36:28 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 13 03:28:24.870384288 (106h2m19.301464419s) [upload=110366]: Starting upload Aug 13 03:28:24.870426058 (106h2m19.30150618s) [upload=110366]: Done starting upload Aug 13 03:28:24.870488848 (106h2m19.301568979s) [upload=110366]: Uploading segments... Aug 13 03:28:24.870604439 (106h2m19.30168457s) [upload=110366]: Got next segment. Inline: false Aug 13 03:28:24.870637743 (106h2m19.301717874s) [upload=110366]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:28:24.870642031 (106h2m19.301722162s) [upload=110366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 13 03:28:25.321382698 (106h2m19.752462879s) [seg_pos={0 0}, upload=110366]: Begin upload segment... Aug 13 03:28:25.321441521 (106h2m19.752521652s) [seg_pos={0 0}, upload=110366]: Done begin upload segment. Aug 13 03:28:25.321465406 (106h2m19.752545527s) [noise=true , address=104.36.82.44:4876 , piece_id=576P5WMAQ6BCLTG4UAJX2ZCHGAMKCD47Z6YQBWJKHI5MQGGXT4XA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.321492969 (106h2m19.75257309s) [noise=true , address=104.36.82.37:4686 , piece_id=TWWJUP6LY7VX4BW7PSTTZK53JTGSVBP2Y6NQCR3PXT44WCLGY2WA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.321866553 (106h2m19.752946694s) [noise=true , address=104.36.82.4:4833 , piece_id=WOJB675NZW5MONNZ7MHECJEYFRFUQDDQFCG4EATF2DTTQOY5FPIA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322159584 (106h2m19.753239715s) [noise=true , address=104.36.82.38:4716 , piece_id=3Q4BIBCFEYXICWG3CEFEUIBAMBNZWJ7FNCMMB4DHJ73RHKVXGBKQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322826730 (106h2m19.753906862s) [noise=true , address=104.36.82.35:4585 , piece_id=EA7XTERIGFLJMV33KWPJJEFXKDNYHSVJIBYES33AANWX6RFO3Z4A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322949396 (106h2m19.754029517s) [noise=true , address=104.36.82.15:4267 , piece_id=YOBGDLVCGO7Z6XLQQ6K7INK36KDGOHZNA6L2ROJF2VVC3YOE5CEQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323182792 (106h2m19.754262923s) [noise=true , address=64.49.13.139:4473 , piece_id=GYBREOGJD222LYFTSGTYHYLM4T22ZKSQYKN4GNW42ZQSNZQOHQJA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323412270 (106h2m19.754492402s) [noise=true , address=104.36.82.60:4363 , piece_id=SRLXSTGJIEBAM4H6UE6AF32SJY7N2ZHJPPDOXKTUN44ZI3LT2I5A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323740879 (106h2m19.75482101s) [noise=true , address=104.36.82.5:4866 , piece_id=22IXIQNEVDD6VILGPIQMS46JFYVQDI3BNI633KABQLCDBQQB22ZA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.324341969 (106h2m19.755422101s) [noise=true , address=104.36.82.46:4496 , piece_id=K6J6AEVYWM63WM6IFDXSGBUWK6WX72R26RLVLRZ64VSF4H2744JA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.324756232 (106h2m19.755836363s) [noise=true , address=104.36.82.8:4988 , piece_id=KZO4HIEGBYBTXIUMP7YETN5WBBNS2ZMGOA3W6H377RF5RV3RIGRQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325088147 (106h2m19.756168278s) [noise=true , address=64.49.13.139:4478 , piece_id=ZLJZ3QKNWE7LHFZSZEFOSZ6EN27KIAJFEEY7TVL4YW2VZXQM6IOA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325336562 (106h2m19.756416693s) [noise=true , address=104.36.82.36:4641 , piece_id=3A5FI2NFX2Y2QYX6JMEMIPXAYU6ZGB662F6PZGGY6W456MNN4H2Q, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325678096 (106h2m19.756758217s) [noise=true , address=104.36.82.39:4758 , piece_id=6GQJIBOJ6XK6RILWWEPLNQH7OPKSYWFMQG3SXYTVONZRLV3YDU7A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325997056 (106h2m19.757077187s) [noise=true , address=79.127.220.98:4422 , piece_id=R6WFYRTY4IEQ4ORVBDWP5FBP676QRCJDPKSBNJDXL64C2OH2SWDA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326218289 (106h2m19.75729842s) [noise=true , address=104.36.82.21:4488 , piece_id=ZJC77OAGWGPQHJLXGSY2JUS4POTFI4UYW2ORN2OET4U5QCCK2RYA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326380920 (106h2m19.757461051s) [noise=true , address=77.90.31.21:4906 , piece_id=FLIZ63F7NOFHDUEDYOAGMMGEGS473UQPXY2C3WKX67MGWJJEZBIA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326717245 (106h2m19.757797376s) [noise=true , address=104.36.82.58:4298 , piece_id=I76VJFWH36OITPI2LEHTUPWFSC566LKQZBGM7VVJ6744TYN2QESA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:26.509048537 (106h2m20.940128668s) [upload=110366]: Next returned nil segment Aug 13 03:28:26.509066711 (106h2m20.940146842s) [upload=110366]: Waiting for error group managing segments... Aug 13 03:28:26.809943799 (106h2m21.24102393s) [noise=true , address=64.49.13.139:4478 , piece_id=ZLJZ3QKNWE7LHFZSZEFOSZ6EN27KIAJFEEY7TVL4YW2VZXQM6IOA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:26.884546048 (106h2m21.315626179s) [noise=true , address=64.49.13.139:4473 , piece_id=GYBREOGJD222LYFTSGTYHYLM4T22ZKSQYKN4GNW42ZQSNZQOHQJA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.268324420 (106h2m21.699404611s) [noise=true , address=104.36.82.38:4716 , piece_id=3Q4BIBCFEYXICWG3CEFEUIBAMBNZWJ7FNCMMB4DHJ73RHKVXGBKQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.288897151 (106h2m21.719977283s) [noise=true , address=104.36.82.58:4298 , piece_id=I76VJFWH36OITPI2LEHTUPWFSC566LKQZBGM7VVJ6744TYN2QESA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.298496038 (106h2m21.729576169s) [noise=true , address=104.36.82.36:4641 , piece_id=3A5FI2NFX2Y2QYX6JMEMIPXAYU6ZGB662F6PZGGY6W456MNN4H2Q, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.305049777 (106h2m21.736129908s) [noise=true , address=104.36.82.35:4585 , piece_id=EA7XTERIGFLJMV33KWPJJEFXKDNYHSVJIBYES33AANWX6RFO3Z4A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.319397510 (106h2m21.750477631s) [noise=true , address=104.36.82.21:4488 , piece_id=ZJC77OAGWGPQHJLXGSY2JUS4POTFI4UYW2ORN2OET4U5QCCK2RYA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.324171042 (106h2m21.755251173s) [noise=true , address=104.36.82.8:4988 , piece_id=KZO4HIEGBYBTXIUMP7YETN5WBBNS2ZMGOA3W6H377RF5RV3RIGRQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.325044093 (106h2m21.756124224s) [noise=true , address=104.36.82.39:4758 , piece_id=6GQJIBOJ6XK6RILWWEPLNQH7OPKSYWFMQG3SXYTVONZRLV3YDU7A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.326189485 (106h2m21.757269616s) [noise=true , address=104.36.82.4:4833 , piece_id=WOJB675NZW5MONNZ7MHECJEYFRFUQDDQFCG4EATF2DTTQOY5FPIA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.358731589 (106h2m21.78981172s) [noise=true , address=104.36.82.15:4267 , piece_id=YOBGDLVCGO7Z6XLQQ6K7INK36KDGOHZNA6L2ROJF2VVC3YOE5CEQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.364728292 (106h2m21.795808413s) [noise=true , address=104.36.82.37:4686 , piece_id=TWWJUP6LY7VX4BW7PSTTZK53JTGSVBP2Y6NQCR3PXT44WCLGY2WA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.386279758 (106h2m21.817359889s) [noise=true , address=77.90.31.21:4906 , piece_id=FLIZ63F7NOFHDUEDYOAGMMGEGS473UQPXY2C3WKX67MGWJJEZBIA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.393790007 (106h2m21.824870158s) [noise=true , address=104.36.82.44:4876 , piece_id=576P5WMAQ6BCLTG4UAJX2ZCHGAMKCD47Z6YQBWJKHI5MQGGXT4XA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.393863398 (106h2m21.824943529s) [seg_pos={0 0}, upload=110366]: Segment reached optimal threshold of 14 pieces. Aug 13 03:28:27.393993957 (106h2m21.825074088s) [noise=true , address=104.36.82.46:4496 , piece_id=K6J6AEVYWM63WM6IFDXSGBUWK6WX72R26RLVLRZ64VSF4H2744JA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394051778 (106h2m21.82513191s) [noise=true , address=104.36.82.5:4866 , piece_id=22IXIQNEVDD6VILGPIQMS46JFYVQDI3BNI633KABQLCDBQQB22ZA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zUcYfJE8xRW2Tow7bGB3PeNrVBUzVWR82JADkVwPdSr4CTPyr): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394083198 (106h2m21.82516333s) [noise=true , address=104.36.82.60:4363 , piece_id=SRLXSTGJIEBAM4H6UE6AF32SJY7N2ZHJPPDOXKTUN44ZI3LT2I5A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7): protocol: expected piece hash; write tcp 188.166.225.170:42920->104.36.82.60:4363: use of closed network connection; write tcp 188.166.225.170:42920->104.36.82.60:4363: use of closed network connection; EOF Aug 13 03:28:27.394101453 (106h2m21.825181574s) [noise=true , address=79.127.220.98:4422 , piece_id=R6WFYRTY4IEQ4ORVBDWP5FBP676QRCJDPKSBNJDXL64C2OH2SWDA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MScXvmPpGZ6HRRwaoBMrZR3TvhDQh4nZkpSYggRQGzP1JbyVE): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394138204 (106h2m21.825218335s) [seg_pos={0 0}, upload=110366]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 13 03:28:27.394212385 (106h2m21.825292516s) [upload=110366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 13 03:28:27.394217095 (106h2m21.825297226s) [upload=110366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:28:27.720702752 (106h2m22.151782883s) [upload=110366]: Done uploading segments... Aug 13 03:28:27.720720255 (106h2m22.151800386s) [upload=110366]: Upload finished. err: <nil> Duration: 5.600771482s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 1.86243529s FAIL (2025-08-13 03:36:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 13 03:06:41.898033585 (105h40m36.329113716s) [upload=110007]: Starting upload Aug 13 03:06:41.898093971 (105h40m36.329174102s) [upload=110007]: Done starting upload Aug 13 03:06:41.898140650 (105h40m36.329220781s) [upload=110007]: Uploading segments... Aug 13 03:06:41.898172331 (105h40m36.329252462s) [upload=110007]: Got next segment. Inline: true Aug 13 03:06:41.898176078 (105h40m36.329256209s) [upload=110007]: Waiting for error group managing segments... Aug 13 03:06:41.898208831 (105h40m36.329288962s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:06:41.898212778 (105h40m36.329292909s) [upload=110007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:06:41.898215563 (105h40m36.329295694s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:07:11.244546166 (105h41m5.675626307s) [upload=110007]: Done uploading segments... Aug 13 03:07:11.244585330 (105h41m5.675665482s) [upload=110007]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.862459035s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 1.897142999s FAIL (2025-08-13 03:36:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 13 03:23:34.987469695 (105h57m29.418549826s) [upload=110284]: Starting upload Aug 13 03:23:34.987520002 (105h57m29.418600133s) [upload=110284]: Done starting upload Aug 13 03:23:34.987570067 (105h57m29.418650198s) [upload=110284]: Uploading segments... Aug 13 03:23:34.987604243 (105h57m29.418684374s) [upload=110284]: Got next segment. Inline: true Aug 13 03:23:34.987608641 (105h57m29.418688772s) [upload=110284]: Waiting for error group managing segments... Aug 13 03:23:34.987640973 (105h57m29.418721094s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:23:34.987644710 (105h57m29.418724841s) [upload=110284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:23:34.987647044 (105h57m29.418727165s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:23:35.199946062 (105h57m29.631026193s) [upload=110284]: Done uploading segments... Aug 13 03:23:35.199964076 (105h57m29.631044207s) [upload=110284]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.897172135s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................X.....XX....................] duration: 7.780235083s FAIL (2025-08-13 03:35:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.780257836s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X...........X...............X...................X.........] duration: 5.628586662s FAIL (2025-08-13 03:35:25 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 13 03:28:24.870384288 (106h2m19.301464419s) [upload=110366]: Starting upload Aug 13 03:28:24.870426058 (106h2m19.30150618s) [upload=110366]: Done starting upload Aug 13 03:28:24.870488848 (106h2m19.301568979s) [upload=110366]: Uploading segments... Aug 13 03:28:24.870604439 (106h2m19.30168457s) [upload=110366]: Got next segment. Inline: false Aug 13 03:28:24.870637743 (106h2m19.301717874s) [upload=110366]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:28:24.870642031 (106h2m19.301722162s) [upload=110366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 13 03:28:25.321382698 (106h2m19.752462879s) [seg_pos={0 0}, upload=110366]: Begin upload segment... Aug 13 03:28:25.321441521 (106h2m19.752521652s) [seg_pos={0 0}, upload=110366]: Done begin upload segment. Aug 13 03:28:25.321465406 (106h2m19.752545527s) [noise=true , address=104.36.82.44:4876 , piece_id=576P5WMAQ6BCLTG4UAJX2ZCHGAMKCD47Z6YQBWJKHI5MQGGXT4XA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.321492969 (106h2m19.75257309s) [noise=true , address=104.36.82.37:4686 , piece_id=TWWJUP6LY7VX4BW7PSTTZK53JTGSVBP2Y6NQCR3PXT44WCLGY2WA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.321866553 (106h2m19.752946694s) [noise=true , address=104.36.82.4:4833 , piece_id=WOJB675NZW5MONNZ7MHECJEYFRFUQDDQFCG4EATF2DTTQOY5FPIA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322159584 (106h2m19.753239715s) [noise=true , address=104.36.82.38:4716 , piece_id=3Q4BIBCFEYXICWG3CEFEUIBAMBNZWJ7FNCMMB4DHJ73RHKVXGBKQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322826730 (106h2m19.753906862s) [noise=true , address=104.36.82.35:4585 , piece_id=EA7XTERIGFLJMV33KWPJJEFXKDNYHSVJIBYES33AANWX6RFO3Z4A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322949396 (106h2m19.754029517s) [noise=true , address=104.36.82.15:4267 , piece_id=YOBGDLVCGO7Z6XLQQ6K7INK36KDGOHZNA6L2ROJF2VVC3YOE5CEQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323182792 (106h2m19.754262923s) [noise=true , address=64.49.13.139:4473 , piece_id=GYBREOGJD222LYFTSGTYHYLM4T22ZKSQYKN4GNW42ZQSNZQOHQJA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323412270 (106h2m19.754492402s) [noise=true , address=104.36.82.60:4363 , piece_id=SRLXSTGJIEBAM4H6UE6AF32SJY7N2ZHJPPDOXKTUN44ZI3LT2I5A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323740879 (106h2m19.75482101s) [noise=true , address=104.36.82.5:4866 , piece_id=22IXIQNEVDD6VILGPIQMS46JFYVQDI3BNI633KABQLCDBQQB22ZA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.324341969 (106h2m19.755422101s) [noise=true , address=104.36.82.46:4496 , piece_id=K6J6AEVYWM63WM6IFDXSGBUWK6WX72R26RLVLRZ64VSF4H2744JA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.324756232 (106h2m19.755836363s) [noise=true , address=104.36.82.8:4988 , piece_id=KZO4HIEGBYBTXIUMP7YETN5WBBNS2ZMGOA3W6H377RF5RV3RIGRQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325088147 (106h2m19.756168278s) [noise=true , address=64.49.13.139:4478 , piece_id=ZLJZ3QKNWE7LHFZSZEFOSZ6EN27KIAJFEEY7TVL4YW2VZXQM6IOA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325336562 (106h2m19.756416693s) [noise=true , address=104.36.82.36:4641 , piece_id=3A5FI2NFX2Y2QYX6JMEMIPXAYU6ZGB662F6PZGGY6W456MNN4H2Q, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325678096 (106h2m19.756758217s) [noise=true , address=104.36.82.39:4758 , piece_id=6GQJIBOJ6XK6RILWWEPLNQH7OPKSYWFMQG3SXYTVONZRLV3YDU7A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325997056 (106h2m19.757077187s) [noise=true , address=79.127.220.98:4422 , piece_id=R6WFYRTY4IEQ4ORVBDWP5FBP676QRCJDPKSBNJDXL64C2OH2SWDA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326218289 (106h2m19.75729842s) [noise=true , address=104.36.82.21:4488 , piece_id=ZJC77OAGWGPQHJLXGSY2JUS4POTFI4UYW2ORN2OET4U5QCCK2RYA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326380920 (106h2m19.757461051s) [noise=true , address=77.90.31.21:4906 , piece_id=FLIZ63F7NOFHDUEDYOAGMMGEGS473UQPXY2C3WKX67MGWJJEZBIA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326717245 (106h2m19.757797376s) [noise=true , address=104.36.82.58:4298 , piece_id=I76VJFWH36OITPI2LEHTUPWFSC566LKQZBGM7VVJ6744TYN2QESA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:26.509048537 (106h2m20.940128668s) [upload=110366]: Next returned nil segment Aug 13 03:28:26.509066711 (106h2m20.940146842s) [upload=110366]: Waiting for error group managing segments... Aug 13 03:28:26.809943799 (106h2m21.24102393s) [noise=true , address=64.49.13.139:4478 , piece_id=ZLJZ3QKNWE7LHFZSZEFOSZ6EN27KIAJFEEY7TVL4YW2VZXQM6IOA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:26.884546048 (106h2m21.315626179s) [noise=true , address=64.49.13.139:4473 , piece_id=GYBREOGJD222LYFTSGTYHYLM4T22ZKSQYKN4GNW42ZQSNZQOHQJA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.268324420 (106h2m21.699404611s) [noise=true , address=104.36.82.38:4716 , piece_id=3Q4BIBCFEYXICWG3CEFEUIBAMBNZWJ7FNCMMB4DHJ73RHKVXGBKQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.288897151 (106h2m21.719977283s) [noise=true , address=104.36.82.58:4298 , piece_id=I76VJFWH36OITPI2LEHTUPWFSC566LKQZBGM7VVJ6744TYN2QESA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.298496038 (106h2m21.729576169s) [noise=true , address=104.36.82.36:4641 , piece_id=3A5FI2NFX2Y2QYX6JMEMIPXAYU6ZGB662F6PZGGY6W456MNN4H2Q, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.305049777 (106h2m21.736129908s) [noise=true , address=104.36.82.35:4585 , piece_id=EA7XTERIGFLJMV33KWPJJEFXKDNYHSVJIBYES33AANWX6RFO3Z4A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.319397510 (106h2m21.750477631s) [noise=true , address=104.36.82.21:4488 , piece_id=ZJC77OAGWGPQHJLXGSY2JUS4POTFI4UYW2ORN2OET4U5QCCK2RYA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.324171042 (106h2m21.755251173s) [noise=true , address=104.36.82.8:4988 , piece_id=KZO4HIEGBYBTXIUMP7YETN5WBBNS2ZMGOA3W6H377RF5RV3RIGRQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.325044093 (106h2m21.756124224s) [noise=true , address=104.36.82.39:4758 , piece_id=6GQJIBOJ6XK6RILWWEPLNQH7OPKSYWFMQG3SXYTVONZRLV3YDU7A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.326189485 (106h2m21.757269616s) [noise=true , address=104.36.82.4:4833 , piece_id=WOJB675NZW5MONNZ7MHECJEYFRFUQDDQFCG4EATF2DTTQOY5FPIA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.358731589 (106h2m21.78981172s) [noise=true , address=104.36.82.15:4267 , piece_id=YOBGDLVCGO7Z6XLQQ6K7INK36KDGOHZNA6L2ROJF2VVC3YOE5CEQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.364728292 (106h2m21.795808413s) [noise=true , address=104.36.82.37:4686 , piece_id=TWWJUP6LY7VX4BW7PSTTZK53JTGSVBP2Y6NQCR3PXT44WCLGY2WA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.386279758 (106h2m21.817359889s) [noise=true , address=77.90.31.21:4906 , piece_id=FLIZ63F7NOFHDUEDYOAGMMGEGS473UQPXY2C3WKX67MGWJJEZBIA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.393790007 (106h2m21.824870158s) [noise=true , address=104.36.82.44:4876 , piece_id=576P5WMAQ6BCLTG4UAJX2ZCHGAMKCD47Z6YQBWJKHI5MQGGXT4XA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.393863398 (106h2m21.824943529s) [seg_pos={0 0}, upload=110366]: Segment reached optimal threshold of 14 pieces. Aug 13 03:28:27.393993957 (106h2m21.825074088s) [noise=true , address=104.36.82.46:4496 , piece_id=K6J6AEVYWM63WM6IFDXSGBUWK6WX72R26RLVLRZ64VSF4H2744JA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394051778 (106h2m21.82513191s) [noise=true , address=104.36.82.5:4866 , piece_id=22IXIQNEVDD6VILGPIQMS46JFYVQDI3BNI633KABQLCDBQQB22ZA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zUcYfJE8xRW2Tow7bGB3PeNrVBUzVWR82JADkVwPdSr4CTPyr): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394083198 (106h2m21.82516333s) [noise=true , address=104.36.82.60:4363 , piece_id=SRLXSTGJIEBAM4H6UE6AF32SJY7N2ZHJPPDOXKTUN44ZI3LT2I5A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7): protocol: expected piece hash; write tcp 188.166.225.170:42920->104.36.82.60:4363: use of closed network connection; write tcp 188.166.225.170:42920->104.36.82.60:4363: use of closed network connection; EOF Aug 13 03:28:27.394101453 (106h2m21.825181574s) [noise=true , address=79.127.220.98:4422 , piece_id=R6WFYRTY4IEQ4ORVBDWP5FBP676QRCJDPKSBNJDXL64C2OH2SWDA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MScXvmPpGZ6HRRwaoBMrZR3TvhDQh4nZkpSYggRQGzP1JbyVE): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394138204 (106h2m21.825218335s) [seg_pos={0 0}, upload=110366]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 13 03:28:27.394212385 (106h2m21.825292516s) [upload=110366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 13 03:28:27.394217095 (106h2m21.825297226s) [upload=110366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:28:27.720702752 (106h2m22.151782883s) [upload=110366]: Done uploading segments... Aug 13 03:28:27.720720255 (106h2m22.151800386s) [upload=110366]: Upload finished. err: <nil> Duration: 5.62860755s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 1.868925587s FAIL (2025-08-13 03:35:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 13 03:23:34.987469695 (105h57m29.418549826s) [upload=110284]: Starting upload Aug 13 03:23:34.987520002 (105h57m29.418600133s) [upload=110284]: Done starting upload Aug 13 03:23:34.987570067 (105h57m29.418650198s) [upload=110284]: Uploading segments... Aug 13 03:23:34.987604243 (105h57m29.418684374s) [upload=110284]: Got next segment. Inline: true Aug 13 03:23:34.987608641 (105h57m29.418688772s) [upload=110284]: Waiting for error group managing segments... Aug 13 03:23:34.987640973 (105h57m29.418721094s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:23:34.987644710 (105h57m29.418724841s) [upload=110284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:23:34.987647044 (105h57m29.418727165s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:23:35.199946062 (105h57m29.631026193s) [upload=110284]: Done uploading segments... Aug 13 03:23:35.199964076 (105h57m29.631044207s) [upload=110284]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.868940235s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 1.891434556s FAIL (2025-08-13 03:35:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 13 03:06:41.898033585 (105h40m36.329113716s) [upload=110007]: Starting upload Aug 13 03:06:41.898093971 (105h40m36.329174102s) [upload=110007]: Done starting upload Aug 13 03:06:41.898140650 (105h40m36.329220781s) [upload=110007]: Uploading segments... Aug 13 03:06:41.898172331 (105h40m36.329252462s) [upload=110007]: Got next segment. Inline: true Aug 13 03:06:41.898176078 (105h40m36.329256209s) [upload=110007]: Waiting for error group managing segments... Aug 13 03:06:41.898208831 (105h40m36.329288962s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:06:41.898212778 (105h40m36.329292909s) [upload=110007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:06:41.898215563 (105h40m36.329295694s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:07:11.244546166 (105h41m5.675626307s) [upload=110007]: Done uploading segments... Aug 13 03:07:11.244585330 (105h41m5.675665482s) [upload=110007]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.891455635s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................X.....XX...................] duration: 7.802702744s FAIL (2025-08-13 03:35:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.802721549s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X...........X...............X...................X........] duration: 5.825148223s FAIL (2025-08-13 03:34:24 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 13 03:28:24.870384288 (106h2m19.301464419s) [upload=110366]: Starting upload Aug 13 03:28:24.870426058 (106h2m19.30150618s) [upload=110366]: Done starting upload Aug 13 03:28:24.870488848 (106h2m19.301568979s) [upload=110366]: Uploading segments... Aug 13 03:28:24.870604439 (106h2m19.30168457s) [upload=110366]: Got next segment. Inline: false Aug 13 03:28:24.870637743 (106h2m19.301717874s) [upload=110366]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:28:24.870642031 (106h2m19.301722162s) [upload=110366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 13 03:28:25.321382698 (106h2m19.752462879s) [seg_pos={0 0}, upload=110366]: Begin upload segment... Aug 13 03:28:25.321441521 (106h2m19.752521652s) [seg_pos={0 0}, upload=110366]: Done begin upload segment. Aug 13 03:28:25.321465406 (106h2m19.752545527s) [noise=true , address=104.36.82.44:4876 , piece_id=576P5WMAQ6BCLTG4UAJX2ZCHGAMKCD47Z6YQBWJKHI5MQGGXT4XA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.321492969 (106h2m19.75257309s) [noise=true , address=104.36.82.37:4686 , piece_id=TWWJUP6LY7VX4BW7PSTTZK53JTGSVBP2Y6NQCR3PXT44WCLGY2WA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.321866553 (106h2m19.752946694s) [noise=true , address=104.36.82.4:4833 , piece_id=WOJB675NZW5MONNZ7MHECJEYFRFUQDDQFCG4EATF2DTTQOY5FPIA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322159584 (106h2m19.753239715s) [noise=true , address=104.36.82.38:4716 , piece_id=3Q4BIBCFEYXICWG3CEFEUIBAMBNZWJ7FNCMMB4DHJ73RHKVXGBKQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322826730 (106h2m19.753906862s) [noise=true , address=104.36.82.35:4585 , piece_id=EA7XTERIGFLJMV33KWPJJEFXKDNYHSVJIBYES33AANWX6RFO3Z4A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322949396 (106h2m19.754029517s) [noise=true , address=104.36.82.15:4267 , piece_id=YOBGDLVCGO7Z6XLQQ6K7INK36KDGOHZNA6L2ROJF2VVC3YOE5CEQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323182792 (106h2m19.754262923s) [noise=true , address=64.49.13.139:4473 , piece_id=GYBREOGJD222LYFTSGTYHYLM4T22ZKSQYKN4GNW42ZQSNZQOHQJA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323412270 (106h2m19.754492402s) [noise=true , address=104.36.82.60:4363 , piece_id=SRLXSTGJIEBAM4H6UE6AF32SJY7N2ZHJPPDOXKTUN44ZI3LT2I5A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323740879 (106h2m19.75482101s) [noise=true , address=104.36.82.5:4866 , piece_id=22IXIQNEVDD6VILGPIQMS46JFYVQDI3BNI633KABQLCDBQQB22ZA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.324341969 (106h2m19.755422101s) [noise=true , address=104.36.82.46:4496 , piece_id=K6J6AEVYWM63WM6IFDXSGBUWK6WX72R26RLVLRZ64VSF4H2744JA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.324756232 (106h2m19.755836363s) [noise=true , address=104.36.82.8:4988 , piece_id=KZO4HIEGBYBTXIUMP7YETN5WBBNS2ZMGOA3W6H377RF5RV3RIGRQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325088147 (106h2m19.756168278s) [noise=true , address=64.49.13.139:4478 , piece_id=ZLJZ3QKNWE7LHFZSZEFOSZ6EN27KIAJFEEY7TVL4YW2VZXQM6IOA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325336562 (106h2m19.756416693s) [noise=true , address=104.36.82.36:4641 , piece_id=3A5FI2NFX2Y2QYX6JMEMIPXAYU6ZGB662F6PZGGY6W456MNN4H2Q, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325678096 (106h2m19.756758217s) [noise=true , address=104.36.82.39:4758 , piece_id=6GQJIBOJ6XK6RILWWEPLNQH7OPKSYWFMQG3SXYTVONZRLV3YDU7A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325997056 (106h2m19.757077187s) [noise=true , address=79.127.220.98:4422 , piece_id=R6WFYRTY4IEQ4ORVBDWP5FBP676QRCJDPKSBNJDXL64C2OH2SWDA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326218289 (106h2m19.75729842s) [noise=true , address=104.36.82.21:4488 , piece_id=ZJC77OAGWGPQHJLXGSY2JUS4POTFI4UYW2ORN2OET4U5QCCK2RYA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326380920 (106h2m19.757461051s) [noise=true , address=77.90.31.21:4906 , piece_id=FLIZ63F7NOFHDUEDYOAGMMGEGS473UQPXY2C3WKX67MGWJJEZBIA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326717245 (106h2m19.757797376s) [noise=true , address=104.36.82.58:4298 , piece_id=I76VJFWH36OITPI2LEHTUPWFSC566LKQZBGM7VVJ6744TYN2QESA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:26.509048537 (106h2m20.940128668s) [upload=110366]: Next returned nil segment Aug 13 03:28:26.509066711 (106h2m20.940146842s) [upload=110366]: Waiting for error group managing segments... Aug 13 03:28:26.809943799 (106h2m21.24102393s) [noise=true , address=64.49.13.139:4478 , piece_id=ZLJZ3QKNWE7LHFZSZEFOSZ6EN27KIAJFEEY7TVL4YW2VZXQM6IOA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:26.884546048 (106h2m21.315626179s) [noise=true , address=64.49.13.139:4473 , piece_id=GYBREOGJD222LYFTSGTYHYLM4T22ZKSQYKN4GNW42ZQSNZQOHQJA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.268324420 (106h2m21.699404611s) [noise=true , address=104.36.82.38:4716 , piece_id=3Q4BIBCFEYXICWG3CEFEUIBAMBNZWJ7FNCMMB4DHJ73RHKVXGBKQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.288897151 (106h2m21.719977283s) [noise=true , address=104.36.82.58:4298 , piece_id=I76VJFWH36OITPI2LEHTUPWFSC566LKQZBGM7VVJ6744TYN2QESA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.298496038 (106h2m21.729576169s) [noise=true , address=104.36.82.36:4641 , piece_id=3A5FI2NFX2Y2QYX6JMEMIPXAYU6ZGB662F6PZGGY6W456MNN4H2Q, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.305049777 (106h2m21.736129908s) [noise=true , address=104.36.82.35:4585 , piece_id=EA7XTERIGFLJMV33KWPJJEFXKDNYHSVJIBYES33AANWX6RFO3Z4A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.319397510 (106h2m21.750477631s) [noise=true , address=104.36.82.21:4488 , piece_id=ZJC77OAGWGPQHJLXGSY2JUS4POTFI4UYW2ORN2OET4U5QCCK2RYA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.324171042 (106h2m21.755251173s) [noise=true , address=104.36.82.8:4988 , piece_id=KZO4HIEGBYBTXIUMP7YETN5WBBNS2ZMGOA3W6H377RF5RV3RIGRQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.325044093 (106h2m21.756124224s) [noise=true , address=104.36.82.39:4758 , piece_id=6GQJIBOJ6XK6RILWWEPLNQH7OPKSYWFMQG3SXYTVONZRLV3YDU7A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.326189485 (106h2m21.757269616s) [noise=true , address=104.36.82.4:4833 , piece_id=WOJB675NZW5MONNZ7MHECJEYFRFUQDDQFCG4EATF2DTTQOY5FPIA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.358731589 (106h2m21.78981172s) [noise=true , address=104.36.82.15:4267 , piece_id=YOBGDLVCGO7Z6XLQQ6K7INK36KDGOHZNA6L2ROJF2VVC3YOE5CEQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.364728292 (106h2m21.795808413s) [noise=true , address=104.36.82.37:4686 , piece_id=TWWJUP6LY7VX4BW7PSTTZK53JTGSVBP2Y6NQCR3PXT44WCLGY2WA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.386279758 (106h2m21.817359889s) [noise=true , address=77.90.31.21:4906 , piece_id=FLIZ63F7NOFHDUEDYOAGMMGEGS473UQPXY2C3WKX67MGWJJEZBIA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.393790007 (106h2m21.824870158s) [noise=true , address=104.36.82.44:4876 , piece_id=576P5WMAQ6BCLTG4UAJX2ZCHGAMKCD47Z6YQBWJKHI5MQGGXT4XA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.393863398 (106h2m21.824943529s) [seg_pos={0 0}, upload=110366]: Segment reached optimal threshold of 14 pieces. Aug 13 03:28:27.393993957 (106h2m21.825074088s) [noise=true , address=104.36.82.46:4496 , piece_id=K6J6AEVYWM63WM6IFDXSGBUWK6WX72R26RLVLRZ64VSF4H2744JA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394051778 (106h2m21.82513191s) [noise=true , address=104.36.82.5:4866 , piece_id=22IXIQNEVDD6VILGPIQMS46JFYVQDI3BNI633KABQLCDBQQB22ZA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zUcYfJE8xRW2Tow7bGB3PeNrVBUzVWR82JADkVwPdSr4CTPyr): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394083198 (106h2m21.82516333s) [noise=true , address=104.36.82.60:4363 , piece_id=SRLXSTGJIEBAM4H6UE6AF32SJY7N2ZHJPPDOXKTUN44ZI3LT2I5A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7): protocol: expected piece hash; write tcp 188.166.225.170:42920->104.36.82.60:4363: use of closed network connection; write tcp 188.166.225.170:42920->104.36.82.60:4363: use of closed network connection; EOF Aug 13 03:28:27.394101453 (106h2m21.825181574s) [noise=true , address=79.127.220.98:4422 , piece_id=R6WFYRTY4IEQ4ORVBDWP5FBP676QRCJDPKSBNJDXL64C2OH2SWDA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MScXvmPpGZ6HRRwaoBMrZR3TvhDQh4nZkpSYggRQGzP1JbyVE): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394138204 (106h2m21.825218335s) [seg_pos={0 0}, upload=110366]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 13 03:28:27.394212385 (106h2m21.825292516s) [upload=110366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 13 03:28:27.394217095 (106h2m21.825297226s) [upload=110366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:28:27.720702752 (106h2m22.151782883s) [upload=110366]: Done uploading segments... Aug 13 03:28:27.720720255 (106h2m22.151800386s) [upload=110366]: Upload finished. err: <nil> Duration: 5.825171256s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................X.....XX..................] duration: 7.736732752s FAIL (2025-08-13 03:34:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.736772388s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 1.839447425s FAIL (2025-08-13 03:34:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 13 03:23:34.987469695 (105h57m29.418549826s) [upload=110284]: Starting upload Aug 13 03:23:34.987520002 (105h57m29.418600133s) [upload=110284]: Done starting upload Aug 13 03:23:34.987570067 (105h57m29.418650198s) [upload=110284]: Uploading segments... Aug 13 03:23:34.987604243 (105h57m29.418684374s) [upload=110284]: Got next segment. Inline: true Aug 13 03:23:34.987608641 (105h57m29.418688772s) [upload=110284]: Waiting for error group managing segments... Aug 13 03:23:34.987640973 (105h57m29.418721094s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:23:34.987644710 (105h57m29.418724841s) [upload=110284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:23:34.987647044 (105h57m29.418727165s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:23:35.199946062 (105h57m29.631026193s) [upload=110284]: Done uploading segments... Aug 13 03:23:35.199964076 (105h57m29.631044207s) [upload=110284]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.839475029s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 1.876834059s FAIL (2025-08-13 03:34:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 13 03:06:41.898033585 (105h40m36.329113716s) [upload=110007]: Starting upload Aug 13 03:06:41.898093971 (105h40m36.329174102s) [upload=110007]: Done starting upload Aug 13 03:06:41.898140650 (105h40m36.329220781s) [upload=110007]: Uploading segments... Aug 13 03:06:41.898172331 (105h40m36.329252462s) [upload=110007]: Got next segment. Inline: true Aug 13 03:06:41.898176078 (105h40m36.329256209s) [upload=110007]: Waiting for error group managing segments... Aug 13 03:06:41.898208831 (105h40m36.329288962s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:06:41.898212778 (105h40m36.329292909s) [upload=110007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:06:41.898215563 (105h40m36.329295694s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:07:11.244546166 (105h41m5.675626307s) [upload=110007]: Done uploading segments... Aug 13 03:07:11.244585330 (105h41m5.675665482s) [upload=110007]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.876862263s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 1.872570744s FAIL (2025-08-13 03:33:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 13 03:23:34.987469695 (105h57m29.418549826s) [upload=110284]: Starting upload Aug 13 03:23:34.987520002 (105h57m29.418600133s) [upload=110284]: Done starting upload Aug 13 03:23:34.987570067 (105h57m29.418650198s) [upload=110284]: Uploading segments... Aug 13 03:23:34.987604243 (105h57m29.418684374s) [upload=110284]: Got next segment. Inline: true Aug 13 03:23:34.987608641 (105h57m29.418688772s) [upload=110284]: Waiting for error group managing segments... Aug 13 03:23:34.987640973 (105h57m29.418721094s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:23:34.987644710 (105h57m29.418724841s) [upload=110284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:23:34.987647044 (105h57m29.418727165s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:23:35.199946062 (105h57m29.631026193s) [upload=110284]: Done uploading segments... Aug 13 03:23:35.199964076 (105h57m29.631044207s) [upload=110284]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.872606262s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X...........X...............X...................X.......] duration: 6.168599335s FAIL (2025-08-13 03:33:41 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 13 03:28:24.870384288 (106h2m19.301464419s) [upload=110366]: Starting upload Aug 13 03:28:24.870426058 (106h2m19.30150618s) [upload=110366]: Done starting upload Aug 13 03:28:24.870488848 (106h2m19.301568979s) [upload=110366]: Uploading segments... Aug 13 03:28:24.870604439 (106h2m19.30168457s) [upload=110366]: Got next segment. Inline: false Aug 13 03:28:24.870637743 (106h2m19.301717874s) [upload=110366]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:28:24.870642031 (106h2m19.301722162s) [upload=110366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 13 03:28:25.321382698 (106h2m19.752462879s) [seg_pos={0 0}, upload=110366]: Begin upload segment... Aug 13 03:28:25.321441521 (106h2m19.752521652s) [seg_pos={0 0}, upload=110366]: Done begin upload segment. Aug 13 03:28:25.321465406 (106h2m19.752545527s) [noise=true , address=104.36.82.44:4876 , piece_id=576P5WMAQ6BCLTG4UAJX2ZCHGAMKCD47Z6YQBWJKHI5MQGGXT4XA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.321492969 (106h2m19.75257309s) [noise=true , address=104.36.82.37:4686 , piece_id=TWWJUP6LY7VX4BW7PSTTZK53JTGSVBP2Y6NQCR3PXT44WCLGY2WA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.321866553 (106h2m19.752946694s) [noise=true , address=104.36.82.4:4833 , piece_id=WOJB675NZW5MONNZ7MHECJEYFRFUQDDQFCG4EATF2DTTQOY5FPIA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322159584 (106h2m19.753239715s) [noise=true , address=104.36.82.38:4716 , piece_id=3Q4BIBCFEYXICWG3CEFEUIBAMBNZWJ7FNCMMB4DHJ73RHKVXGBKQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322826730 (106h2m19.753906862s) [noise=true , address=104.36.82.35:4585 , piece_id=EA7XTERIGFLJMV33KWPJJEFXKDNYHSVJIBYES33AANWX6RFO3Z4A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322949396 (106h2m19.754029517s) [noise=true , address=104.36.82.15:4267 , piece_id=YOBGDLVCGO7Z6XLQQ6K7INK36KDGOHZNA6L2ROJF2VVC3YOE5CEQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323182792 (106h2m19.754262923s) [noise=true , address=64.49.13.139:4473 , piece_id=GYBREOGJD222LYFTSGTYHYLM4T22ZKSQYKN4GNW42ZQSNZQOHQJA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323412270 (106h2m19.754492402s) [noise=true , address=104.36.82.60:4363 , piece_id=SRLXSTGJIEBAM4H6UE6AF32SJY7N2ZHJPPDOXKTUN44ZI3LT2I5A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323740879 (106h2m19.75482101s) [noise=true , address=104.36.82.5:4866 , piece_id=22IXIQNEVDD6VILGPIQMS46JFYVQDI3BNI633KABQLCDBQQB22ZA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.324341969 (106h2m19.755422101s) [noise=true , address=104.36.82.46:4496 , piece_id=K6J6AEVYWM63WM6IFDXSGBUWK6WX72R26RLVLRZ64VSF4H2744JA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.324756232 (106h2m19.755836363s) [noise=true , address=104.36.82.8:4988 , piece_id=KZO4HIEGBYBTXIUMP7YETN5WBBNS2ZMGOA3W6H377RF5RV3RIGRQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325088147 (106h2m19.756168278s) [noise=true , address=64.49.13.139:4478 , piece_id=ZLJZ3QKNWE7LHFZSZEFOSZ6EN27KIAJFEEY7TVL4YW2VZXQM6IOA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325336562 (106h2m19.756416693s) [noise=true , address=104.36.82.36:4641 , piece_id=3A5FI2NFX2Y2QYX6JMEMIPXAYU6ZGB662F6PZGGY6W456MNN4H2Q, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325678096 (106h2m19.756758217s) [noise=true , address=104.36.82.39:4758 , piece_id=6GQJIBOJ6XK6RILWWEPLNQH7OPKSYWFMQG3SXYTVONZRLV3YDU7A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325997056 (106h2m19.757077187s) [noise=true , address=79.127.220.98:4422 , piece_id=R6WFYRTY4IEQ4ORVBDWP5FBP676QRCJDPKSBNJDXL64C2OH2SWDA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326218289 (106h2m19.75729842s) [noise=true , address=104.36.82.21:4488 , piece_id=ZJC77OAGWGPQHJLXGSY2JUS4POTFI4UYW2ORN2OET4U5QCCK2RYA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326380920 (106h2m19.757461051s) [noise=true , address=77.90.31.21:4906 , piece_id=FLIZ63F7NOFHDUEDYOAGMMGEGS473UQPXY2C3WKX67MGWJJEZBIA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326717245 (106h2m19.757797376s) [noise=true , address=104.36.82.58:4298 , piece_id=I76VJFWH36OITPI2LEHTUPWFSC566LKQZBGM7VVJ6744TYN2QESA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:26.509048537 (106h2m20.940128668s) [upload=110366]: Next returned nil segment Aug 13 03:28:26.509066711 (106h2m20.940146842s) [upload=110366]: Waiting for error group managing segments... Aug 13 03:28:26.809943799 (106h2m21.24102393s) [noise=true , address=64.49.13.139:4478 , piece_id=ZLJZ3QKNWE7LHFZSZEFOSZ6EN27KIAJFEEY7TVL4YW2VZXQM6IOA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:26.884546048 (106h2m21.315626179s) [noise=true , address=64.49.13.139:4473 , piece_id=GYBREOGJD222LYFTSGTYHYLM4T22ZKSQYKN4GNW42ZQSNZQOHQJA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.268324420 (106h2m21.699404611s) [noise=true , address=104.36.82.38:4716 , piece_id=3Q4BIBCFEYXICWG3CEFEUIBAMBNZWJ7FNCMMB4DHJ73RHKVXGBKQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.288897151 (106h2m21.719977283s) [noise=true , address=104.36.82.58:4298 , piece_id=I76VJFWH36OITPI2LEHTUPWFSC566LKQZBGM7VVJ6744TYN2QESA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.298496038 (106h2m21.729576169s) [noise=true , address=104.36.82.36:4641 , piece_id=3A5FI2NFX2Y2QYX6JMEMIPXAYU6ZGB662F6PZGGY6W456MNN4H2Q, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.305049777 (106h2m21.736129908s) [noise=true , address=104.36.82.35:4585 , piece_id=EA7XTERIGFLJMV33KWPJJEFXKDNYHSVJIBYES33AANWX6RFO3Z4A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.319397510 (106h2m21.750477631s) [noise=true , address=104.36.82.21:4488 , piece_id=ZJC77OAGWGPQHJLXGSY2JUS4POTFI4UYW2ORN2OET4U5QCCK2RYA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.324171042 (106h2m21.755251173s) [noise=true , address=104.36.82.8:4988 , piece_id=KZO4HIEGBYBTXIUMP7YETN5WBBNS2ZMGOA3W6H377RF5RV3RIGRQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.325044093 (106h2m21.756124224s) [noise=true , address=104.36.82.39:4758 , piece_id=6GQJIBOJ6XK6RILWWEPLNQH7OPKSYWFMQG3SXYTVONZRLV3YDU7A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.326189485 (106h2m21.757269616s) [noise=true , address=104.36.82.4:4833 , piece_id=WOJB675NZW5MONNZ7MHECJEYFRFUQDDQFCG4EATF2DTTQOY5FPIA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.358731589 (106h2m21.78981172s) [noise=true , address=104.36.82.15:4267 , piece_id=YOBGDLVCGO7Z6XLQQ6K7INK36KDGOHZNA6L2ROJF2VVC3YOE5CEQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.364728292 (106h2m21.795808413s) [noise=true , address=104.36.82.37:4686 , piece_id=TWWJUP6LY7VX4BW7PSTTZK53JTGSVBP2Y6NQCR3PXT44WCLGY2WA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.386279758 (106h2m21.817359889s) [noise=true , address=77.90.31.21:4906 , piece_id=FLIZ63F7NOFHDUEDYOAGMMGEGS473UQPXY2C3WKX67MGWJJEZBIA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.393790007 (106h2m21.824870158s) [noise=true , address=104.36.82.44:4876 , piece_id=576P5WMAQ6BCLTG4UAJX2ZCHGAMKCD47Z6YQBWJKHI5MQGGXT4XA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.393863398 (106h2m21.824943529s) [seg_pos={0 0}, upload=110366]: Segment reached optimal threshold of 14 pieces. Aug 13 03:28:27.393993957 (106h2m21.825074088s) [noise=true , address=104.36.82.46:4496 , piece_id=K6J6AEVYWM63WM6IFDXSGBUWK6WX72R26RLVLRZ64VSF4H2744JA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394051778 (106h2m21.82513191s) [noise=true , address=104.36.82.5:4866 , piece_id=22IXIQNEVDD6VILGPIQMS46JFYVQDI3BNI633KABQLCDBQQB22ZA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zUcYfJE8xRW2Tow7bGB3PeNrVBUzVWR82JADkVwPdSr4CTPyr): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394083198 (106h2m21.82516333s) [noise=true , address=104.36.82.60:4363 , piece_id=SRLXSTGJIEBAM4H6UE6AF32SJY7N2ZHJPPDOXKTUN44ZI3LT2I5A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7): protocol: expected piece hash; write tcp 188.166.225.170:42920->104.36.82.60:4363: use of closed network connection; write tcp 188.166.225.170:42920->104.36.82.60:4363: use of closed network connection; EOF Aug 13 03:28:27.394101453 (106h2m21.825181574s) [noise=true , address=79.127.220.98:4422 , piece_id=R6WFYRTY4IEQ4ORVBDWP5FBP676QRCJDPKSBNJDXL64C2OH2SWDA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MScXvmPpGZ6HRRwaoBMrZR3TvhDQh4nZkpSYggRQGzP1JbyVE): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394138204 (106h2m21.825218335s) [seg_pos={0 0}, upload=110366]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 13 03:28:27.394212385 (106h2m21.825292516s) [upload=110366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 13 03:28:27.394217095 (106h2m21.825297226s) [upload=110366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:28:27.720702752 (106h2m22.151782883s) [upload=110366]: Done uploading segments... Aug 13 03:28:27.720720255 (106h2m22.151800386s) [upload=110366]: Upload finished. err: <nil> Duration: 6.168617821s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 1.872736298s FAIL (2025-08-13 03:33:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 13 03:06:41.898033585 (105h40m36.329113716s) [upload=110007]: Starting upload Aug 13 03:06:41.898093971 (105h40m36.329174102s) [upload=110007]: Done starting upload Aug 13 03:06:41.898140650 (105h40m36.329220781s) [upload=110007]: Uploading segments... Aug 13 03:06:41.898172331 (105h40m36.329252462s) [upload=110007]: Got next segment. Inline: true Aug 13 03:06:41.898176078 (105h40m36.329256209s) [upload=110007]: Waiting for error group managing segments... Aug 13 03:06:41.898208831 (105h40m36.329288962s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:06:41.898212778 (105h40m36.329292909s) [upload=110007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:06:41.898215563 (105h40m36.329295694s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:07:11.244546166 (105h41m5.675626307s) [upload=110007]: Done uploading segments... Aug 13 03:07:11.244585330 (105h41m5.675665482s) [upload=110007]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.872764182s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 1.845755457s FAIL (2025-08-13 03:33:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 13 03:23:34.987469695 (105h57m29.418549826s) [upload=110284]: Starting upload Aug 13 03:23:34.987520002 (105h57m29.418600133s) [upload=110284]: Done starting upload Aug 13 03:23:34.987570067 (105h57m29.418650198s) [upload=110284]: Uploading segments... Aug 13 03:23:34.987604243 (105h57m29.418684374s) [upload=110284]: Got next segment. Inline: true Aug 13 03:23:34.987608641 (105h57m29.418688772s) [upload=110284]: Waiting for error group managing segments... Aug 13 03:23:34.987640973 (105h57m29.418721094s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:23:34.987644710 (105h57m29.418724841s) [upload=110284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:23:34.987647044 (105h57m29.418727165s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:23:35.199946062 (105h57m29.631026193s) [upload=110284]: Done uploading segments... Aug 13 03:23:35.199964076 (105h57m29.631044207s) [upload=110284]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.845787057s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................X.....XX.................] duration: 11.269388562s FAIL (2025-08-13 03:33:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.269408961s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.110283326s FAIL (2025-08-13 03:32:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 13 03:23:34.987469695 (105h57m29.418549826s) [upload=110284]: Starting upload Aug 13 03:23:34.987520002 (105h57m29.418600133s) [upload=110284]: Done starting upload Aug 13 03:23:34.987570067 (105h57m29.418650198s) [upload=110284]: Uploading segments... Aug 13 03:23:34.987604243 (105h57m29.418684374s) [upload=110284]: Got next segment. Inline: true Aug 13 03:23:34.987608641 (105h57m29.418688772s) [upload=110284]: Waiting for error group managing segments... Aug 13 03:23:34.987640973 (105h57m29.418721094s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:23:34.987644710 (105h57m29.418724841s) [upload=110284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:23:34.987647044 (105h57m29.418727165s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:23:35.199946062 (105h57m29.631026193s) [upload=110284]: Done uploading segments... Aug 13 03:23:35.199964076 (105h57m29.631044207s) [upload=110284]: Upload finished. err: metaclient: Too Many Requests Duration:* 2.110311631s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 1.914378986s FAIL (2025-08-13 03:32:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 13 03:06:41.898033585 (105h40m36.329113716s) [upload=110007]: Starting upload Aug 13 03:06:41.898093971 (105h40m36.329174102s) [upload=110007]: Done starting upload Aug 13 03:06:41.898140650 (105h40m36.329220781s) [upload=110007]: Uploading segments... Aug 13 03:06:41.898172331 (105h40m36.329252462s) [upload=110007]: Got next segment. Inline: true Aug 13 03:06:41.898176078 (105h40m36.329256209s) [upload=110007]: Waiting for error group managing segments... Aug 13 03:06:41.898208831 (105h40m36.329288962s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:06:41.898212778 (105h40m36.329292909s) [upload=110007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:06:41.898215563 (105h40m36.329295694s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:07:11.244546166 (105h41m5.675626307s) [upload=110007]: Done uploading segments... Aug 13 03:07:11.244585330 (105h41m5.675665482s) [upload=110007]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.914409704s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X...........X...............X...................X......] duration: 6.108615498s FAIL (2025-08-13 03:32:42 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 13 03:28:24.870384288 (106h2m19.301464419s) [upload=110366]: Starting upload Aug 13 03:28:24.870426058 (106h2m19.30150618s) [upload=110366]: Done starting upload Aug 13 03:28:24.870488848 (106h2m19.301568979s) [upload=110366]: Uploading segments... Aug 13 03:28:24.870604439 (106h2m19.30168457s) [upload=110366]: Got next segment. Inline: false Aug 13 03:28:24.870637743 (106h2m19.301717874s) [upload=110366]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:28:24.870642031 (106h2m19.301722162s) [upload=110366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 13 03:28:25.321382698 (106h2m19.752462879s) [seg_pos={0 0}, upload=110366]: Begin upload segment... Aug 13 03:28:25.321441521 (106h2m19.752521652s) [seg_pos={0 0}, upload=110366]: Done begin upload segment. Aug 13 03:28:25.321465406 (106h2m19.752545527s) [noise=true , address=104.36.82.44:4876 , piece_id=576P5WMAQ6BCLTG4UAJX2ZCHGAMKCD47Z6YQBWJKHI5MQGGXT4XA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.321492969 (106h2m19.75257309s) [noise=true , address=104.36.82.37:4686 , piece_id=TWWJUP6LY7VX4BW7PSTTZK53JTGSVBP2Y6NQCR3PXT44WCLGY2WA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.321866553 (106h2m19.752946694s) [noise=true , address=104.36.82.4:4833 , piece_id=WOJB675NZW5MONNZ7MHECJEYFRFUQDDQFCG4EATF2DTTQOY5FPIA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322159584 (106h2m19.753239715s) [noise=true , address=104.36.82.38:4716 , piece_id=3Q4BIBCFEYXICWG3CEFEUIBAMBNZWJ7FNCMMB4DHJ73RHKVXGBKQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322826730 (106h2m19.753906862s) [noise=true , address=104.36.82.35:4585 , piece_id=EA7XTERIGFLJMV33KWPJJEFXKDNYHSVJIBYES33AANWX6RFO3Z4A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322949396 (106h2m19.754029517s) [noise=true , address=104.36.82.15:4267 , piece_id=YOBGDLVCGO7Z6XLQQ6K7INK36KDGOHZNA6L2ROJF2VVC3YOE5CEQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323182792 (106h2m19.754262923s) [noise=true , address=64.49.13.139:4473 , piece_id=GYBREOGJD222LYFTSGTYHYLM4T22ZKSQYKN4GNW42ZQSNZQOHQJA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323412270 (106h2m19.754492402s) [noise=true , address=104.36.82.60:4363 , piece_id=SRLXSTGJIEBAM4H6UE6AF32SJY7N2ZHJPPDOXKTUN44ZI3LT2I5A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323740879 (106h2m19.75482101s) [noise=true , address=104.36.82.5:4866 , piece_id=22IXIQNEVDD6VILGPIQMS46JFYVQDI3BNI633KABQLCDBQQB22ZA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.324341969 (106h2m19.755422101s) [noise=true , address=104.36.82.46:4496 , piece_id=K6J6AEVYWM63WM6IFDXSGBUWK6WX72R26RLVLRZ64VSF4H2744JA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.324756232 (106h2m19.755836363s) [noise=true , address=104.36.82.8:4988 , piece_id=KZO4HIEGBYBTXIUMP7YETN5WBBNS2ZMGOA3W6H377RF5RV3RIGRQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325088147 (106h2m19.756168278s) [noise=true , address=64.49.13.139:4478 , piece_id=ZLJZ3QKNWE7LHFZSZEFOSZ6EN27KIAJFEEY7TVL4YW2VZXQM6IOA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325336562 (106h2m19.756416693s) [noise=true , address=104.36.82.36:4641 , piece_id=3A5FI2NFX2Y2QYX6JMEMIPXAYU6ZGB662F6PZGGY6W456MNN4H2Q, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325678096 (106h2m19.756758217s) [noise=true , address=104.36.82.39:4758 , piece_id=6GQJIBOJ6XK6RILWWEPLNQH7OPKSYWFMQG3SXYTVONZRLV3YDU7A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325997056 (106h2m19.757077187s) [noise=true , address=79.127.220.98:4422 , piece_id=R6WFYRTY4IEQ4ORVBDWP5FBP676QRCJDPKSBNJDXL64C2OH2SWDA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326218289 (106h2m19.75729842s) [noise=true , address=104.36.82.21:4488 , piece_id=ZJC77OAGWGPQHJLXGSY2JUS4POTFI4UYW2ORN2OET4U5QCCK2RYA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326380920 (106h2m19.757461051s) [noise=true , address=77.90.31.21:4906 , piece_id=FLIZ63F7NOFHDUEDYOAGMMGEGS473UQPXY2C3WKX67MGWJJEZBIA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326717245 (106h2m19.757797376s) [noise=true , address=104.36.82.58:4298 , piece_id=I76VJFWH36OITPI2LEHTUPWFSC566LKQZBGM7VVJ6744TYN2QESA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:26.509048537 (106h2m20.940128668s) [upload=110366]: Next returned nil segment Aug 13 03:28:26.509066711 (106h2m20.940146842s) [upload=110366]: Waiting for error group managing segments... Aug 13 03:28:26.809943799 (106h2m21.24102393s) [noise=true , address=64.49.13.139:4478 , piece_id=ZLJZ3QKNWE7LHFZSZEFOSZ6EN27KIAJFEEY7TVL4YW2VZXQM6IOA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:26.884546048 (106h2m21.315626179s) [noise=true , address=64.49.13.139:4473 , piece_id=GYBREOGJD222LYFTSGTYHYLM4T22ZKSQYKN4GNW42ZQSNZQOHQJA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.268324420 (106h2m21.699404611s) [noise=true , address=104.36.82.38:4716 , piece_id=3Q4BIBCFEYXICWG3CEFEUIBAMBNZWJ7FNCMMB4DHJ73RHKVXGBKQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.288897151 (106h2m21.719977283s) [noise=true , address=104.36.82.58:4298 , piece_id=I76VJFWH36OITPI2LEHTUPWFSC566LKQZBGM7VVJ6744TYN2QESA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.298496038 (106h2m21.729576169s) [noise=true , address=104.36.82.36:4641 , piece_id=3A5FI2NFX2Y2QYX6JMEMIPXAYU6ZGB662F6PZGGY6W456MNN4H2Q, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.305049777 (106h2m21.736129908s) [noise=true , address=104.36.82.35:4585 , piece_id=EA7XTERIGFLJMV33KWPJJEFXKDNYHSVJIBYES33AANWX6RFO3Z4A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.319397510 (106h2m21.750477631s) [noise=true , address=104.36.82.21:4488 , piece_id=ZJC77OAGWGPQHJLXGSY2JUS4POTFI4UYW2ORN2OET4U5QCCK2RYA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.324171042 (106h2m21.755251173s) [noise=true , address=104.36.82.8:4988 , piece_id=KZO4HIEGBYBTXIUMP7YETN5WBBNS2ZMGOA3W6H377RF5RV3RIGRQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.325044093 (106h2m21.756124224s) [noise=true , address=104.36.82.39:4758 , piece_id=6GQJIBOJ6XK6RILWWEPLNQH7OPKSYWFMQG3SXYTVONZRLV3YDU7A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.326189485 (106h2m21.757269616s) [noise=true , address=104.36.82.4:4833 , piece_id=WOJB675NZW5MONNZ7MHECJEYFRFUQDDQFCG4EATF2DTTQOY5FPIA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.358731589 (106h2m21.78981172s) [noise=true , address=104.36.82.15:4267 , piece_id=YOBGDLVCGO7Z6XLQQ6K7INK36KDGOHZNA6L2ROJF2VVC3YOE5CEQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.364728292 (106h2m21.795808413s) [noise=true , address=104.36.82.37:4686 , piece_id=TWWJUP6LY7VX4BW7PSTTZK53JTGSVBP2Y6NQCR3PXT44WCLGY2WA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.386279758 (106h2m21.817359889s) [noise=true , address=77.90.31.21:4906 , piece_id=FLIZ63F7NOFHDUEDYOAGMMGEGS473UQPXY2C3WKX67MGWJJEZBIA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.393790007 (106h2m21.824870158s) [noise=true , address=104.36.82.44:4876 , piece_id=576P5WMAQ6BCLTG4UAJX2ZCHGAMKCD47Z6YQBWJKHI5MQGGXT4XA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.393863398 (106h2m21.824943529s) [seg_pos={0 0}, upload=110366]: Segment reached optimal threshold of 14 pieces. Aug 13 03:28:27.393993957 (106h2m21.825074088s) [noise=true , address=104.36.82.46:4496 , piece_id=K6J6AEVYWM63WM6IFDXSGBUWK6WX72R26RLVLRZ64VSF4H2744JA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394051778 (106h2m21.82513191s) [noise=true , address=104.36.82.5:4866 , piece_id=22IXIQNEVDD6VILGPIQMS46JFYVQDI3BNI633KABQLCDBQQB22ZA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zUcYfJE8xRW2Tow7bGB3PeNrVBUzVWR82JADkVwPdSr4CTPyr): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394083198 (106h2m21.82516333s) [noise=true , address=104.36.82.60:4363 , piece_id=SRLXSTGJIEBAM4H6UE6AF32SJY7N2ZHJPPDOXKTUN44ZI3LT2I5A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7): protocol: expected piece hash; write tcp 188.166.225.170:42920->104.36.82.60:4363: use of closed network connection; write tcp 188.166.225.170:42920->104.36.82.60:4363: use of closed network connection; EOF Aug 13 03:28:27.394101453 (106h2m21.825181574s) [noise=true , address=79.127.220.98:4422 , piece_id=R6WFYRTY4IEQ4ORVBDWP5FBP676QRCJDPKSBNJDXL64C2OH2SWDA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MScXvmPpGZ6HRRwaoBMrZR3TvhDQh4nZkpSYggRQGzP1JbyVE): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394138204 (106h2m21.825218335s) [seg_pos={0 0}, upload=110366]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 13 03:28:27.394212385 (106h2m21.825292516s) [upload=110366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 13 03:28:27.394217095 (106h2m21.825297226s) [upload=110366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:28:27.720702752 (106h2m22.151782883s) [upload=110366]: Done uploading segments... Aug 13 03:28:27.720720255 (106h2m22.151800386s) [upload=110366]: Upload finished. err: <nil> Duration: 6.108637672s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X...........X...............X...................X.....] duration: 5.616768628s FAIL (2025-08-13 03:32:19 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 13 03:28:24.870384288 (106h2m19.301464419s) [upload=110366]: Starting upload Aug 13 03:28:24.870426058 (106h2m19.30150618s) [upload=110366]: Done starting upload Aug 13 03:28:24.870488848 (106h2m19.301568979s) [upload=110366]: Uploading segments... Aug 13 03:28:24.870604439 (106h2m19.30168457s) [upload=110366]: Got next segment. Inline: false Aug 13 03:28:24.870637743 (106h2m19.301717874s) [upload=110366]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:28:24.870642031 (106h2m19.301722162s) [upload=110366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 13 03:28:25.321382698 (106h2m19.752462879s) [seg_pos={0 0}, upload=110366]: Begin upload segment... Aug 13 03:28:25.321441521 (106h2m19.752521652s) [seg_pos={0 0}, upload=110366]: Done begin upload segment. Aug 13 03:28:25.321465406 (106h2m19.752545527s) [noise=true , address=104.36.82.44:4876 , piece_id=576P5WMAQ6BCLTG4UAJX2ZCHGAMKCD47Z6YQBWJKHI5MQGGXT4XA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.321492969 (106h2m19.75257309s) [noise=true , address=104.36.82.37:4686 , piece_id=TWWJUP6LY7VX4BW7PSTTZK53JTGSVBP2Y6NQCR3PXT44WCLGY2WA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.321866553 (106h2m19.752946694s) [noise=true , address=104.36.82.4:4833 , piece_id=WOJB675NZW5MONNZ7MHECJEYFRFUQDDQFCG4EATF2DTTQOY5FPIA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322159584 (106h2m19.753239715s) [noise=true , address=104.36.82.38:4716 , piece_id=3Q4BIBCFEYXICWG3CEFEUIBAMBNZWJ7FNCMMB4DHJ73RHKVXGBKQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322826730 (106h2m19.753906862s) [noise=true , address=104.36.82.35:4585 , piece_id=EA7XTERIGFLJMV33KWPJJEFXKDNYHSVJIBYES33AANWX6RFO3Z4A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322949396 (106h2m19.754029517s) [noise=true , address=104.36.82.15:4267 , piece_id=YOBGDLVCGO7Z6XLQQ6K7INK36KDGOHZNA6L2ROJF2VVC3YOE5CEQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323182792 (106h2m19.754262923s) [noise=true , address=64.49.13.139:4473 , piece_id=GYBREOGJD222LYFTSGTYHYLM4T22ZKSQYKN4GNW42ZQSNZQOHQJA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323412270 (106h2m19.754492402s) [noise=true , address=104.36.82.60:4363 , piece_id=SRLXSTGJIEBAM4H6UE6AF32SJY7N2ZHJPPDOXKTUN44ZI3LT2I5A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323740879 (106h2m19.75482101s) [noise=true , address=104.36.82.5:4866 , piece_id=22IXIQNEVDD6VILGPIQMS46JFYVQDI3BNI633KABQLCDBQQB22ZA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.324341969 (106h2m19.755422101s) [noise=true , address=104.36.82.46:4496 , piece_id=K6J6AEVYWM63WM6IFDXSGBUWK6WX72R26RLVLRZ64VSF4H2744JA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.324756232 (106h2m19.755836363s) [noise=true , address=104.36.82.8:4988 , piece_id=KZO4HIEGBYBTXIUMP7YETN5WBBNS2ZMGOA3W6H377RF5RV3RIGRQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325088147 (106h2m19.756168278s) [noise=true , address=64.49.13.139:4478 , piece_id=ZLJZ3QKNWE7LHFZSZEFOSZ6EN27KIAJFEEY7TVL4YW2VZXQM6IOA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325336562 (106h2m19.756416693s) [noise=true , address=104.36.82.36:4641 , piece_id=3A5FI2NFX2Y2QYX6JMEMIPXAYU6ZGB662F6PZGGY6W456MNN4H2Q, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325678096 (106h2m19.756758217s) [noise=true , address=104.36.82.39:4758 , piece_id=6GQJIBOJ6XK6RILWWEPLNQH7OPKSYWFMQG3SXYTVONZRLV3YDU7A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325997056 (106h2m19.757077187s) [noise=true , address=79.127.220.98:4422 , piece_id=R6WFYRTY4IEQ4ORVBDWP5FBP676QRCJDPKSBNJDXL64C2OH2SWDA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326218289 (106h2m19.75729842s) [noise=true , address=104.36.82.21:4488 , piece_id=ZJC77OAGWGPQHJLXGSY2JUS4POTFI4UYW2ORN2OET4U5QCCK2RYA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326380920 (106h2m19.757461051s) [noise=true , address=77.90.31.21:4906 , piece_id=FLIZ63F7NOFHDUEDYOAGMMGEGS473UQPXY2C3WKX67MGWJJEZBIA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326717245 (106h2m19.757797376s) [noise=true , address=104.36.82.58:4298 , piece_id=I76VJFWH36OITPI2LEHTUPWFSC566LKQZBGM7VVJ6744TYN2QESA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:26.509048537 (106h2m20.940128668s) [upload=110366]: Next returned nil segment Aug 13 03:28:26.509066711 (106h2m20.940146842s) [upload=110366]: Waiting for error group managing segments... Aug 13 03:28:26.809943799 (106h2m21.24102393s) [noise=true , address=64.49.13.139:4478 , piece_id=ZLJZ3QKNWE7LHFZSZEFOSZ6EN27KIAJFEEY7TVL4YW2VZXQM6IOA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:26.884546048 (106h2m21.315626179s) [noise=true , address=64.49.13.139:4473 , piece_id=GYBREOGJD222LYFTSGTYHYLM4T22ZKSQYKN4GNW42ZQSNZQOHQJA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.268324420 (106h2m21.699404611s) [noise=true , address=104.36.82.38:4716 , piece_id=3Q4BIBCFEYXICWG3CEFEUIBAMBNZWJ7FNCMMB4DHJ73RHKVXGBKQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.288897151 (106h2m21.719977283s) [noise=true , address=104.36.82.58:4298 , piece_id=I76VJFWH36OITPI2LEHTUPWFSC566LKQZBGM7VVJ6744TYN2QESA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.298496038 (106h2m21.729576169s) [noise=true , address=104.36.82.36:4641 , piece_id=3A5FI2NFX2Y2QYX6JMEMIPXAYU6ZGB662F6PZGGY6W456MNN4H2Q, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.305049777 (106h2m21.736129908s) [noise=true , address=104.36.82.35:4585 , piece_id=EA7XTERIGFLJMV33KWPJJEFXKDNYHSVJIBYES33AANWX6RFO3Z4A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.319397510 (106h2m21.750477631s) [noise=true , address=104.36.82.21:4488 , piece_id=ZJC77OAGWGPQHJLXGSY2JUS4POTFI4UYW2ORN2OET4U5QCCK2RYA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.324171042 (106h2m21.755251173s) [noise=true , address=104.36.82.8:4988 , piece_id=KZO4HIEGBYBTXIUMP7YETN5WBBNS2ZMGOA3W6H377RF5RV3RIGRQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.325044093 (106h2m21.756124224s) [noise=true , address=104.36.82.39:4758 , piece_id=6GQJIBOJ6XK6RILWWEPLNQH7OPKSYWFMQG3SXYTVONZRLV3YDU7A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.326189485 (106h2m21.757269616s) [noise=true , address=104.36.82.4:4833 , piece_id=WOJB675NZW5MONNZ7MHECJEYFRFUQDDQFCG4EATF2DTTQOY5FPIA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.358731589 (106h2m21.78981172s) [noise=true , address=104.36.82.15:4267 , piece_id=YOBGDLVCGO7Z6XLQQ6K7INK36KDGOHZNA6L2ROJF2VVC3YOE5CEQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.364728292 (106h2m21.795808413s) [noise=true , address=104.36.82.37:4686 , piece_id=TWWJUP6LY7VX4BW7PSTTZK53JTGSVBP2Y6NQCR3PXT44WCLGY2WA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.386279758 (106h2m21.817359889s) [noise=true , address=77.90.31.21:4906 , piece_id=FLIZ63F7NOFHDUEDYOAGMMGEGS473UQPXY2C3WKX67MGWJJEZBIA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.393790007 (106h2m21.824870158s) [noise=true , address=104.36.82.44:4876 , piece_id=576P5WMAQ6BCLTG4UAJX2ZCHGAMKCD47Z6YQBWJKHI5MQGGXT4XA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.393863398 (106h2m21.824943529s) [seg_pos={0 0}, upload=110366]: Segment reached optimal threshold of 14 pieces. Aug 13 03:28:27.393993957 (106h2m21.825074088s) [noise=true , address=104.36.82.46:4496 , piece_id=K6J6AEVYWM63WM6IFDXSGBUWK6WX72R26RLVLRZ64VSF4H2744JA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394051778 (106h2m21.82513191s) [noise=true , address=104.36.82.5:4866 , piece_id=22IXIQNEVDD6VILGPIQMS46JFYVQDI3BNI633KABQLCDBQQB22ZA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zUcYfJE8xRW2Tow7bGB3PeNrVBUzVWR82JADkVwPdSr4CTPyr): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394083198 (106h2m21.82516333s) [noise=true , address=104.36.82.60:4363 , piece_id=SRLXSTGJIEBAM4H6UE6AF32SJY7N2ZHJPPDOXKTUN44ZI3LT2I5A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7): protocol: expected piece hash; write tcp 188.166.225.170:42920->104.36.82.60:4363: use of closed network connection; write tcp 188.166.225.170:42920->104.36.82.60:4363: use of closed network connection; EOF Aug 13 03:28:27.394101453 (106h2m21.825181574s) [noise=true , address=79.127.220.98:4422 , piece_id=R6WFYRTY4IEQ4ORVBDWP5FBP676QRCJDPKSBNJDXL64C2OH2SWDA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MScXvmPpGZ6HRRwaoBMrZR3TvhDQh4nZkpSYggRQGzP1JbyVE): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394138204 (106h2m21.825218335s) [seg_pos={0 0}, upload=110366]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 13 03:28:27.394212385 (106h2m21.825292516s) [upload=110366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 13 03:28:27.394217095 (106h2m21.825297226s) [upload=110366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:28:27.720702752 (106h2m22.151782883s) [upload=110366]: Done uploading segments... Aug 13 03:28:27.720720255 (106h2m22.151800386s) [upload=110366]: Upload finished. err: <nil> Duration: 5.616794559s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................X.....XX................] duration: 7.149961651s FAIL (2025-08-13 03:32:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.149983994s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 1.999841011s FAIL (2025-08-13 03:32:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 13 03:23:34.987469695 (105h57m29.418549826s) [upload=110284]: Starting upload Aug 13 03:23:34.987520002 (105h57m29.418600133s) [upload=110284]: Done starting upload Aug 13 03:23:34.987570067 (105h57m29.418650198s) [upload=110284]: Uploading segments... Aug 13 03:23:34.987604243 (105h57m29.418684374s) [upload=110284]: Got next segment. Inline: true Aug 13 03:23:34.987608641 (105h57m29.418688772s) [upload=110284]: Waiting for error group managing segments... Aug 13 03:23:34.987640973 (105h57m29.418721094s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:23:34.987644710 (105h57m29.418724841s) [upload=110284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:23:34.987647044 (105h57m29.418727165s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:23:35.199946062 (105h57m29.631026193s) [upload=110284]: Done uploading segments... Aug 13 03:23:35.199964076 (105h57m29.631044207s) [upload=110284]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.999864105s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 1.900470373s FAIL (2025-08-13 03:32:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 13 03:06:41.898033585 (105h40m36.329113716s) [upload=110007]: Starting upload Aug 13 03:06:41.898093971 (105h40m36.329174102s) [upload=110007]: Done starting upload Aug 13 03:06:41.898140650 (105h40m36.329220781s) [upload=110007]: Uploading segments... Aug 13 03:06:41.898172331 (105h40m36.329252462s) [upload=110007]: Got next segment. Inline: true Aug 13 03:06:41.898176078 (105h40m36.329256209s) [upload=110007]: Waiting for error group managing segments... Aug 13 03:06:41.898208831 (105h40m36.329288962s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:06:41.898212778 (105h40m36.329292909s) [upload=110007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:06:41.898215563 (105h40m36.329295694s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:07:11.244546166 (105h41m5.675626307s) [upload=110007]: Done uploading segments... Aug 13 03:07:11.244585330 (105h41m5.675665482s) [upload=110007]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.900492235s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 1.892981903s FAIL (2025-08-13 03:31:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 13 03:06:41.898033585 (105h40m36.329113716s) [upload=110007]: Starting upload Aug 13 03:06:41.898093971 (105h40m36.329174102s) [upload=110007]: Done starting upload Aug 13 03:06:41.898140650 (105h40m36.329220781s) [upload=110007]: Uploading segments... Aug 13 03:06:41.898172331 (105h40m36.329252462s) [upload=110007]: Got next segment. Inline: true Aug 13 03:06:41.898176078 (105h40m36.329256209s) [upload=110007]: Waiting for error group managing segments... Aug 13 03:06:41.898208831 (105h40m36.329288962s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:06:41.898212778 (105h40m36.329292909s) [upload=110007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:06:41.898215563 (105h40m36.329295694s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:07:11.244546166 (105h41m5.675626307s) [upload=110007]: Done uploading segments... Aug 13 03:07:11.244585330 (105h41m5.675665482s) [upload=110007]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.893006961s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X...........X...............X...................X....] duration: 5.948721426s FAIL (2025-08-13 03:31:27 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 13 03:28:24.870384288 (106h2m19.301464419s) [upload=110366]: Starting upload Aug 13 03:28:24.870426058 (106h2m19.30150618s) [upload=110366]: Done starting upload Aug 13 03:28:24.870488848 (106h2m19.301568979s) [upload=110366]: Uploading segments... Aug 13 03:28:24.870604439 (106h2m19.30168457s) [upload=110366]: Got next segment. Inline: false Aug 13 03:28:24.870637743 (106h2m19.301717874s) [upload=110366]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:28:24.870642031 (106h2m19.301722162s) [upload=110366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 13 03:28:25.321382698 (106h2m19.752462879s) [seg_pos={0 0}, upload=110366]: Begin upload segment... Aug 13 03:28:25.321441521 (106h2m19.752521652s) [seg_pos={0 0}, upload=110366]: Done begin upload segment. Aug 13 03:28:25.321465406 (106h2m19.752545527s) [noise=true , address=104.36.82.44:4876 , piece_id=576P5WMAQ6BCLTG4UAJX2ZCHGAMKCD47Z6YQBWJKHI5MQGGXT4XA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.321492969 (106h2m19.75257309s) [noise=true , address=104.36.82.37:4686 , piece_id=TWWJUP6LY7VX4BW7PSTTZK53JTGSVBP2Y6NQCR3PXT44WCLGY2WA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.321866553 (106h2m19.752946694s) [noise=true , address=104.36.82.4:4833 , piece_id=WOJB675NZW5MONNZ7MHECJEYFRFUQDDQFCG4EATF2DTTQOY5FPIA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322159584 (106h2m19.753239715s) [noise=true , address=104.36.82.38:4716 , piece_id=3Q4BIBCFEYXICWG3CEFEUIBAMBNZWJ7FNCMMB4DHJ73RHKVXGBKQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322826730 (106h2m19.753906862s) [noise=true , address=104.36.82.35:4585 , piece_id=EA7XTERIGFLJMV33KWPJJEFXKDNYHSVJIBYES33AANWX6RFO3Z4A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322949396 (106h2m19.754029517s) [noise=true , address=104.36.82.15:4267 , piece_id=YOBGDLVCGO7Z6XLQQ6K7INK36KDGOHZNA6L2ROJF2VVC3YOE5CEQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323182792 (106h2m19.754262923s) [noise=true , address=64.49.13.139:4473 , piece_id=GYBREOGJD222LYFTSGTYHYLM4T22ZKSQYKN4GNW42ZQSNZQOHQJA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323412270 (106h2m19.754492402s) [noise=true , address=104.36.82.60:4363 , piece_id=SRLXSTGJIEBAM4H6UE6AF32SJY7N2ZHJPPDOXKTUN44ZI3LT2I5A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323740879 (106h2m19.75482101s) [noise=true , address=104.36.82.5:4866 , piece_id=22IXIQNEVDD6VILGPIQMS46JFYVQDI3BNI633KABQLCDBQQB22ZA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.324341969 (106h2m19.755422101s) [noise=true , address=104.36.82.46:4496 , piece_id=K6J6AEVYWM63WM6IFDXSGBUWK6WX72R26RLVLRZ64VSF4H2744JA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.324756232 (106h2m19.755836363s) [noise=true , address=104.36.82.8:4988 , piece_id=KZO4HIEGBYBTXIUMP7YETN5WBBNS2ZMGOA3W6H377RF5RV3RIGRQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325088147 (106h2m19.756168278s) [noise=true , address=64.49.13.139:4478 , piece_id=ZLJZ3QKNWE7LHFZSZEFOSZ6EN27KIAJFEEY7TVL4YW2VZXQM6IOA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325336562 (106h2m19.756416693s) [noise=true , address=104.36.82.36:4641 , piece_id=3A5FI2NFX2Y2QYX6JMEMIPXAYU6ZGB662F6PZGGY6W456MNN4H2Q, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325678096 (106h2m19.756758217s) [noise=true , address=104.36.82.39:4758 , piece_id=6GQJIBOJ6XK6RILWWEPLNQH7OPKSYWFMQG3SXYTVONZRLV3YDU7A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325997056 (106h2m19.757077187s) [noise=true , address=79.127.220.98:4422 , piece_id=R6WFYRTY4IEQ4ORVBDWP5FBP676QRCJDPKSBNJDXL64C2OH2SWDA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326218289 (106h2m19.75729842s) [noise=true , address=104.36.82.21:4488 , piece_id=ZJC77OAGWGPQHJLXGSY2JUS4POTFI4UYW2ORN2OET4U5QCCK2RYA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326380920 (106h2m19.757461051s) [noise=true , address=77.90.31.21:4906 , piece_id=FLIZ63F7NOFHDUEDYOAGMMGEGS473UQPXY2C3WKX67MGWJJEZBIA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326717245 (106h2m19.757797376s) [noise=true , address=104.36.82.58:4298 , piece_id=I76VJFWH36OITPI2LEHTUPWFSC566LKQZBGM7VVJ6744TYN2QESA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:26.509048537 (106h2m20.940128668s) [upload=110366]: Next returned nil segment Aug 13 03:28:26.509066711 (106h2m20.940146842s) [upload=110366]: Waiting for error group managing segments... Aug 13 03:28:26.809943799 (106h2m21.24102393s) [noise=true , address=64.49.13.139:4478 , piece_id=ZLJZ3QKNWE7LHFZSZEFOSZ6EN27KIAJFEEY7TVL4YW2VZXQM6IOA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:26.884546048 (106h2m21.315626179s) [noise=true , address=64.49.13.139:4473 , piece_id=GYBREOGJD222LYFTSGTYHYLM4T22ZKSQYKN4GNW42ZQSNZQOHQJA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.268324420 (106h2m21.699404611s) [noise=true , address=104.36.82.38:4716 , piece_id=3Q4BIBCFEYXICWG3CEFEUIBAMBNZWJ7FNCMMB4DHJ73RHKVXGBKQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.288897151 (106h2m21.719977283s) [noise=true , address=104.36.82.58:4298 , piece_id=I76VJFWH36OITPI2LEHTUPWFSC566LKQZBGM7VVJ6744TYN2QESA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.298496038 (106h2m21.729576169s) [noise=true , address=104.36.82.36:4641 , piece_id=3A5FI2NFX2Y2QYX6JMEMIPXAYU6ZGB662F6PZGGY6W456MNN4H2Q, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.305049777 (106h2m21.736129908s) [noise=true , address=104.36.82.35:4585 , piece_id=EA7XTERIGFLJMV33KWPJJEFXKDNYHSVJIBYES33AANWX6RFO3Z4A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.319397510 (106h2m21.750477631s) [noise=true , address=104.36.82.21:4488 , piece_id=ZJC77OAGWGPQHJLXGSY2JUS4POTFI4UYW2ORN2OET4U5QCCK2RYA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.324171042 (106h2m21.755251173s) [noise=true , address=104.36.82.8:4988 , piece_id=KZO4HIEGBYBTXIUMP7YETN5WBBNS2ZMGOA3W6H377RF5RV3RIGRQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.325044093 (106h2m21.756124224s) [noise=true , address=104.36.82.39:4758 , piece_id=6GQJIBOJ6XK6RILWWEPLNQH7OPKSYWFMQG3SXYTVONZRLV3YDU7A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.326189485 (106h2m21.757269616s) [noise=true , address=104.36.82.4:4833 , piece_id=WOJB675NZW5MONNZ7MHECJEYFRFUQDDQFCG4EATF2DTTQOY5FPIA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.358731589 (106h2m21.78981172s) [noise=true , address=104.36.82.15:4267 , piece_id=YOBGDLVCGO7Z6XLQQ6K7INK36KDGOHZNA6L2ROJF2VVC3YOE5CEQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.364728292 (106h2m21.795808413s) [noise=true , address=104.36.82.37:4686 , piece_id=TWWJUP6LY7VX4BW7PSTTZK53JTGSVBP2Y6NQCR3PXT44WCLGY2WA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.386279758 (106h2m21.817359889s) [noise=true , address=77.90.31.21:4906 , piece_id=FLIZ63F7NOFHDUEDYOAGMMGEGS473UQPXY2C3WKX67MGWJJEZBIA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.393790007 (106h2m21.824870158s) [noise=true , address=104.36.82.44:4876 , piece_id=576P5WMAQ6BCLTG4UAJX2ZCHGAMKCD47Z6YQBWJKHI5MQGGXT4XA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.393863398 (106h2m21.824943529s) [seg_pos={0 0}, upload=110366]: Segment reached optimal threshold of 14 pieces. Aug 13 03:28:27.393993957 (106h2m21.825074088s) [noise=true , address=104.36.82.46:4496 , piece_id=K6J6AEVYWM63WM6IFDXSGBUWK6WX72R26RLVLRZ64VSF4H2744JA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394051778 (106h2m21.82513191s) [noise=true , address=104.36.82.5:4866 , piece_id=22IXIQNEVDD6VILGPIQMS46JFYVQDI3BNI633KABQLCDBQQB22ZA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zUcYfJE8xRW2Tow7bGB3PeNrVBUzVWR82JADkVwPdSr4CTPyr): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394083198 (106h2m21.82516333s) [noise=true , address=104.36.82.60:4363 , piece_id=SRLXSTGJIEBAM4H6UE6AF32SJY7N2ZHJPPDOXKTUN44ZI3LT2I5A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7): protocol: expected piece hash; write tcp 188.166.225.170:42920->104.36.82.60:4363: use of closed network connection; write tcp 188.166.225.170:42920->104.36.82.60:4363: use of closed network connection; EOF Aug 13 03:28:27.394101453 (106h2m21.825181574s) [noise=true , address=79.127.220.98:4422 , piece_id=R6WFYRTY4IEQ4ORVBDWP5FBP676QRCJDPKSBNJDXL64C2OH2SWDA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MScXvmPpGZ6HRRwaoBMrZR3TvhDQh4nZkpSYggRQGzP1JbyVE): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394138204 (106h2m21.825218335s) [seg_pos={0 0}, upload=110366]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 13 03:28:27.394212385 (106h2m21.825292516s) [upload=110366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 13 03:28:27.394217095 (106h2m21.825297226s) [upload=110366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:28:27.720702752 (106h2m22.151782883s) [upload=110366]: Done uploading segments... Aug 13 03:28:27.720720255 (106h2m22.151800386s) [upload=110366]: Upload finished. err: <nil> Duration: 5.948753747s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................X.....XX...............] duration: 7.773613454s FAIL (2025-08-13 03:31:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.773627871s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 1.878313059s FAIL (2025-08-13 03:31:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 13 03:06:41.898033585 (105h40m36.329113716s) [upload=110007]: Starting upload Aug 13 03:06:41.898093971 (105h40m36.329174102s) [upload=110007]: Done starting upload Aug 13 03:06:41.898140650 (105h40m36.329220781s) [upload=110007]: Uploading segments... Aug 13 03:06:41.898172331 (105h40m36.329252462s) [upload=110007]: Got next segment. Inline: true Aug 13 03:06:41.898176078 (105h40m36.329256209s) [upload=110007]: Waiting for error group managing segments... Aug 13 03:06:41.898208831 (105h40m36.329288962s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:06:41.898212778 (105h40m36.329292909s) [upload=110007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:06:41.898215563 (105h40m36.329295694s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:07:11.244546166 (105h41m5.675626307s) [upload=110007]: Done uploading segments... Aug 13 03:07:11.244585330 (105h41m5.675665482s) [upload=110007]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.878341132s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................X.....XX..............] duration: 9.958468079s FAIL (2025-08-13 03:31:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.958545688s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)