Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X...........X...............X...................X....] duration: 5.948721426s FAIL (2025-08-13 03:31:27 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 13 03:28:24.870384288 (106h2m19.301464419s) [upload=110366]: Starting upload Aug 13 03:28:24.870426058 (106h2m19.30150618s) [upload=110366]: Done starting upload Aug 13 03:28:24.870488848 (106h2m19.301568979s) [upload=110366]: Uploading segments... Aug 13 03:28:24.870604439 (106h2m19.30168457s) [upload=110366]: Got next segment. Inline: false Aug 13 03:28:24.870637743 (106h2m19.301717874s) [upload=110366]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:28:24.870642031 (106h2m19.301722162s) [upload=110366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 13 03:28:25.321382698 (106h2m19.752462879s) [seg_pos={0 0}, upload=110366]: Begin upload segment... Aug 13 03:28:25.321441521 (106h2m19.752521652s) [seg_pos={0 0}, upload=110366]: Done begin upload segment. Aug 13 03:28:25.321465406 (106h2m19.752545527s) [noise=true , address=104.36.82.44:4876 , piece_id=576P5WMAQ6BCLTG4UAJX2ZCHGAMKCD47Z6YQBWJKHI5MQGGXT4XA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.321492969 (106h2m19.75257309s) [noise=true , address=104.36.82.37:4686 , piece_id=TWWJUP6LY7VX4BW7PSTTZK53JTGSVBP2Y6NQCR3PXT44WCLGY2WA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.321866553 (106h2m19.752946694s) [noise=true , address=104.36.82.4:4833 , piece_id=WOJB675NZW5MONNZ7MHECJEYFRFUQDDQFCG4EATF2DTTQOY5FPIA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322159584 (106h2m19.753239715s) [noise=true , address=104.36.82.38:4716 , piece_id=3Q4BIBCFEYXICWG3CEFEUIBAMBNZWJ7FNCMMB4DHJ73RHKVXGBKQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322826730 (106h2m19.753906862s) [noise=true , address=104.36.82.35:4585 , piece_id=EA7XTERIGFLJMV33KWPJJEFXKDNYHSVJIBYES33AANWX6RFO3Z4A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322949396 (106h2m19.754029517s) [noise=true , address=104.36.82.15:4267 , piece_id=YOBGDLVCGO7Z6XLQQ6K7INK36KDGOHZNA6L2ROJF2VVC3YOE5CEQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323182792 (106h2m19.754262923s) [noise=true , address=64.49.13.139:4473 , piece_id=GYBREOGJD222LYFTSGTYHYLM4T22ZKSQYKN4GNW42ZQSNZQOHQJA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323412270 (106h2m19.754492402s) [noise=true , address=104.36.82.60:4363 , piece_id=SRLXSTGJIEBAM4H6UE6AF32SJY7N2ZHJPPDOXKTUN44ZI3LT2I5A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323740879 (106h2m19.75482101s) [noise=true , address=104.36.82.5:4866 , piece_id=22IXIQNEVDD6VILGPIQMS46JFYVQDI3BNI633KABQLCDBQQB22ZA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.324341969 (106h2m19.755422101s) [noise=true , address=104.36.82.46:4496 , piece_id=K6J6AEVYWM63WM6IFDXSGBUWK6WX72R26RLVLRZ64VSF4H2744JA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.324756232 (106h2m19.755836363s) [noise=true , address=104.36.82.8:4988 , piece_id=KZO4HIEGBYBTXIUMP7YETN5WBBNS2ZMGOA3W6H377RF5RV3RIGRQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325088147 (106h2m19.756168278s) [noise=true , address=64.49.13.139:4478 , piece_id=ZLJZ3QKNWE7LHFZSZEFOSZ6EN27KIAJFEEY7TVL4YW2VZXQM6IOA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325336562 (106h2m19.756416693s) [noise=true , address=104.36.82.36:4641 , piece_id=3A5FI2NFX2Y2QYX6JMEMIPXAYU6ZGB662F6PZGGY6W456MNN4H2Q, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325678096 (106h2m19.756758217s) [noise=true , address=104.36.82.39:4758 , piece_id=6GQJIBOJ6XK6RILWWEPLNQH7OPKSYWFMQG3SXYTVONZRLV3YDU7A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325997056 (106h2m19.757077187s) [noise=true , address=79.127.220.98:4422 , piece_id=R6WFYRTY4IEQ4ORVBDWP5FBP676QRCJDPKSBNJDXL64C2OH2SWDA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326218289 (106h2m19.75729842s) [noise=true , address=104.36.82.21:4488 , piece_id=ZJC77OAGWGPQHJLXGSY2JUS4POTFI4UYW2ORN2OET4U5QCCK2RYA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326380920 (106h2m19.757461051s) [noise=true , address=77.90.31.21:4906 , piece_id=FLIZ63F7NOFHDUEDYOAGMMGEGS473UQPXY2C3WKX67MGWJJEZBIA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326717245 (106h2m19.757797376s) [noise=true , address=104.36.82.58:4298 , piece_id=I76VJFWH36OITPI2LEHTUPWFSC566LKQZBGM7VVJ6744TYN2QESA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:26.509048537 (106h2m20.940128668s) [upload=110366]: Next returned nil segment Aug 13 03:28:26.509066711 (106h2m20.940146842s) [upload=110366]: Waiting for error group managing segments... Aug 13 03:28:26.809943799 (106h2m21.24102393s) [noise=true , address=64.49.13.139:4478 , piece_id=ZLJZ3QKNWE7LHFZSZEFOSZ6EN27KIAJFEEY7TVL4YW2VZXQM6IOA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:26.884546048 (106h2m21.315626179s) [noise=true , address=64.49.13.139:4473 , piece_id=GYBREOGJD222LYFTSGTYHYLM4T22ZKSQYKN4GNW42ZQSNZQOHQJA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.268324420 (106h2m21.699404611s) [noise=true , address=104.36.82.38:4716 , piece_id=3Q4BIBCFEYXICWG3CEFEUIBAMBNZWJ7FNCMMB4DHJ73RHKVXGBKQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.288897151 (106h2m21.719977283s) [noise=true , address=104.36.82.58:4298 , piece_id=I76VJFWH36OITPI2LEHTUPWFSC566LKQZBGM7VVJ6744TYN2QESA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.298496038 (106h2m21.729576169s) [noise=true , address=104.36.82.36:4641 , piece_id=3A5FI2NFX2Y2QYX6JMEMIPXAYU6ZGB662F6PZGGY6W456MNN4H2Q, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.305049777 (106h2m21.736129908s) [noise=true , address=104.36.82.35:4585 , piece_id=EA7XTERIGFLJMV33KWPJJEFXKDNYHSVJIBYES33AANWX6RFO3Z4A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.319397510 (106h2m21.750477631s) [noise=true , address=104.36.82.21:4488 , piece_id=ZJC77OAGWGPQHJLXGSY2JUS4POTFI4UYW2ORN2OET4U5QCCK2RYA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.324171042 (106h2m21.755251173s) [noise=true , address=104.36.82.8:4988 , piece_id=KZO4HIEGBYBTXIUMP7YETN5WBBNS2ZMGOA3W6H377RF5RV3RIGRQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.325044093 (106h2m21.756124224s) [noise=true , address=104.36.82.39:4758 , piece_id=6GQJIBOJ6XK6RILWWEPLNQH7OPKSYWFMQG3SXYTVONZRLV3YDU7A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.326189485 (106h2m21.757269616s) [noise=true , address=104.36.82.4:4833 , piece_id=WOJB675NZW5MONNZ7MHECJEYFRFUQDDQFCG4EATF2DTTQOY5FPIA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.358731589 (106h2m21.78981172s) [noise=true , address=104.36.82.15:4267 , piece_id=YOBGDLVCGO7Z6XLQQ6K7INK36KDGOHZNA6L2ROJF2VVC3YOE5CEQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.364728292 (106h2m21.795808413s) [noise=true , address=104.36.82.37:4686 , piece_id=TWWJUP6LY7VX4BW7PSTTZK53JTGSVBP2Y6NQCR3PXT44WCLGY2WA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.386279758 (106h2m21.817359889s) [noise=true , address=77.90.31.21:4906 , piece_id=FLIZ63F7NOFHDUEDYOAGMMGEGS473UQPXY2C3WKX67MGWJJEZBIA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.393790007 (106h2m21.824870158s) [noise=true , address=104.36.82.44:4876 , piece_id=576P5WMAQ6BCLTG4UAJX2ZCHGAMKCD47Z6YQBWJKHI5MQGGXT4XA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.393863398 (106h2m21.824943529s) [seg_pos={0 0}, upload=110366]: Segment reached optimal threshold of 14 pieces. Aug 13 03:28:27.393993957 (106h2m21.825074088s) [noise=true , address=104.36.82.46:4496 , piece_id=K6J6AEVYWM63WM6IFDXSGBUWK6WX72R26RLVLRZ64VSF4H2744JA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394051778 (106h2m21.82513191s) [noise=true , address=104.36.82.5:4866 , piece_id=22IXIQNEVDD6VILGPIQMS46JFYVQDI3BNI633KABQLCDBQQB22ZA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zUcYfJE8xRW2Tow7bGB3PeNrVBUzVWR82JADkVwPdSr4CTPyr): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394083198 (106h2m21.82516333s) [noise=true , address=104.36.82.60:4363 , piece_id=SRLXSTGJIEBAM4H6UE6AF32SJY7N2ZHJPPDOXKTUN44ZI3LT2I5A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7): protocol: expected piece hash; write tcp 188.166.225.170:42920->104.36.82.60:4363: use of closed network connection; write tcp 188.166.225.170:42920->104.36.82.60:4363: use of closed network connection; EOF Aug 13 03:28:27.394101453 (106h2m21.825181574s) [noise=true , address=79.127.220.98:4422 , piece_id=R6WFYRTY4IEQ4ORVBDWP5FBP676QRCJDPKSBNJDXL64C2OH2SWDA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MScXvmPpGZ6HRRwaoBMrZR3TvhDQh4nZkpSYggRQGzP1JbyVE): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394138204 (106h2m21.825218335s) [seg_pos={0 0}, upload=110366]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 13 03:28:27.394212385 (106h2m21.825292516s) [upload=110366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 13 03:28:27.394217095 (106h2m21.825297226s) [upload=110366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:28:27.720702752 (106h2m22.151782883s) [upload=110366]: Done uploading segments... Aug 13 03:28:27.720720255 (106h2m22.151800386s) [upload=110366]: Upload finished. err: <nil> Duration: 5.948753747s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................X.....XX...............] duration: 7.773613454s FAIL (2025-08-13 03:31:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.773627871s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 1.878313059s FAIL (2025-08-13 03:31:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 13 03:06:41.898033585 (105h40m36.329113716s) [upload=110007]: Starting upload Aug 13 03:06:41.898093971 (105h40m36.329174102s) [upload=110007]: Done starting upload Aug 13 03:06:41.898140650 (105h40m36.329220781s) [upload=110007]: Uploading segments... Aug 13 03:06:41.898172331 (105h40m36.329252462s) [upload=110007]: Got next segment. Inline: true Aug 13 03:06:41.898176078 (105h40m36.329256209s) [upload=110007]: Waiting for error group managing segments... Aug 13 03:06:41.898208831 (105h40m36.329288962s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:06:41.898212778 (105h40m36.329292909s) [upload=110007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:06:41.898215563 (105h40m36.329295694s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:07:11.244546166 (105h41m5.675626307s) [upload=110007]: Done uploading segments... Aug 13 03:07:11.244585330 (105h41m5.675665482s) [upload=110007]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.878341132s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................X.....XX..............] duration: 9.958468079s FAIL (2025-08-13 03:31:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.958545688s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 1.969953057s FAIL (2025-08-13 03:30:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 13 03:23:34.987469695 (105h57m29.418549826s) [upload=110284]: Starting upload Aug 13 03:23:34.987520002 (105h57m29.418600133s) [upload=110284]: Done starting upload Aug 13 03:23:34.987570067 (105h57m29.418650198s) [upload=110284]: Uploading segments... Aug 13 03:23:34.987604243 (105h57m29.418684374s) [upload=110284]: Got next segment. Inline: true Aug 13 03:23:34.987608641 (105h57m29.418688772s) [upload=110284]: Waiting for error group managing segments... Aug 13 03:23:34.987640973 (105h57m29.418721094s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:23:34.987644710 (105h57m29.418724841s) [upload=110284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:23:34.987647044 (105h57m29.418727165s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:23:35.199946062 (105h57m29.631026193s) [upload=110284]: Done uploading segments... Aug 13 03:23:35.199964076 (105h57m29.631044207s) [upload=110284]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.969988686s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [X.......X...........X...............X...................X...] duration: 5.757857769s FAIL (2025-08-13 03:30:49 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 13 03:28:24.870384288 (106h2m19.301464419s) [upload=110366]: Starting upload Aug 13 03:28:24.870426058 (106h2m19.30150618s) [upload=110366]: Done starting upload Aug 13 03:28:24.870488848 (106h2m19.301568979s) [upload=110366]: Uploading segments... Aug 13 03:28:24.870604439 (106h2m19.30168457s) [upload=110366]: Got next segment. Inline: false Aug 13 03:28:24.870637743 (106h2m19.301717874s) [upload=110366]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:28:24.870642031 (106h2m19.301722162s) [upload=110366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 13 03:28:25.321382698 (106h2m19.752462879s) [seg_pos={0 0}, upload=110366]: Begin upload segment... Aug 13 03:28:25.321441521 (106h2m19.752521652s) [seg_pos={0 0}, upload=110366]: Done begin upload segment. Aug 13 03:28:25.321465406 (106h2m19.752545527s) [noise=true , address=104.36.82.44:4876 , piece_id=576P5WMAQ6BCLTG4UAJX2ZCHGAMKCD47Z6YQBWJKHI5MQGGXT4XA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.321492969 (106h2m19.75257309s) [noise=true , address=104.36.82.37:4686 , piece_id=TWWJUP6LY7VX4BW7PSTTZK53JTGSVBP2Y6NQCR3PXT44WCLGY2WA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.321866553 (106h2m19.752946694s) [noise=true , address=104.36.82.4:4833 , piece_id=WOJB675NZW5MONNZ7MHECJEYFRFUQDDQFCG4EATF2DTTQOY5FPIA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322159584 (106h2m19.753239715s) [noise=true , address=104.36.82.38:4716 , piece_id=3Q4BIBCFEYXICWG3CEFEUIBAMBNZWJ7FNCMMB4DHJ73RHKVXGBKQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322826730 (106h2m19.753906862s) [noise=true , address=104.36.82.35:4585 , piece_id=EA7XTERIGFLJMV33KWPJJEFXKDNYHSVJIBYES33AANWX6RFO3Z4A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322949396 (106h2m19.754029517s) [noise=true , address=104.36.82.15:4267 , piece_id=YOBGDLVCGO7Z6XLQQ6K7INK36KDGOHZNA6L2ROJF2VVC3YOE5CEQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323182792 (106h2m19.754262923s) [noise=true , address=64.49.13.139:4473 , piece_id=GYBREOGJD222LYFTSGTYHYLM4T22ZKSQYKN4GNW42ZQSNZQOHQJA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323412270 (106h2m19.754492402s) [noise=true , address=104.36.82.60:4363 , piece_id=SRLXSTGJIEBAM4H6UE6AF32SJY7N2ZHJPPDOXKTUN44ZI3LT2I5A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323740879 (106h2m19.75482101s) [noise=true , address=104.36.82.5:4866 , piece_id=22IXIQNEVDD6VILGPIQMS46JFYVQDI3BNI633KABQLCDBQQB22ZA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.324341969 (106h2m19.755422101s) [noise=true , address=104.36.82.46:4496 , piece_id=K6J6AEVYWM63WM6IFDXSGBUWK6WX72R26RLVLRZ64VSF4H2744JA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.324756232 (106h2m19.755836363s) [noise=true , address=104.36.82.8:4988 , piece_id=KZO4HIEGBYBTXIUMP7YETN5WBBNS2ZMGOA3W6H377RF5RV3RIGRQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325088147 (106h2m19.756168278s) [noise=true , address=64.49.13.139:4478 , piece_id=ZLJZ3QKNWE7LHFZSZEFOSZ6EN27KIAJFEEY7TVL4YW2VZXQM6IOA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325336562 (106h2m19.756416693s) [noise=true , address=104.36.82.36:4641 , piece_id=3A5FI2NFX2Y2QYX6JMEMIPXAYU6ZGB662F6PZGGY6W456MNN4H2Q, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325678096 (106h2m19.756758217s) [noise=true , address=104.36.82.39:4758 , piece_id=6GQJIBOJ6XK6RILWWEPLNQH7OPKSYWFMQG3SXYTVONZRLV3YDU7A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325997056 (106h2m19.757077187s) [noise=true , address=79.127.220.98:4422 , piece_id=R6WFYRTY4IEQ4ORVBDWP5FBP676QRCJDPKSBNJDXL64C2OH2SWDA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326218289 (106h2m19.75729842s) [noise=true , address=104.36.82.21:4488 , piece_id=ZJC77OAGWGPQHJLXGSY2JUS4POTFI4UYW2ORN2OET4U5QCCK2RYA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326380920 (106h2m19.757461051s) [noise=true , address=77.90.31.21:4906 , piece_id=FLIZ63F7NOFHDUEDYOAGMMGEGS473UQPXY2C3WKX67MGWJJEZBIA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326717245 (106h2m19.757797376s) [noise=true , address=104.36.82.58:4298 , piece_id=I76VJFWH36OITPI2LEHTUPWFSC566LKQZBGM7VVJ6744TYN2QESA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:26.509048537 (106h2m20.940128668s) [upload=110366]: Next returned nil segment Aug 13 03:28:26.509066711 (106h2m20.940146842s) [upload=110366]: Waiting for error group managing segments... Aug 13 03:28:26.809943799 (106h2m21.24102393s) [noise=true , address=64.49.13.139:4478 , piece_id=ZLJZ3QKNWE7LHFZSZEFOSZ6EN27KIAJFEEY7TVL4YW2VZXQM6IOA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:26.884546048 (106h2m21.315626179s) [noise=true , address=64.49.13.139:4473 , piece_id=GYBREOGJD222LYFTSGTYHYLM4T22ZKSQYKN4GNW42ZQSNZQOHQJA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.268324420 (106h2m21.699404611s) [noise=true , address=104.36.82.38:4716 , piece_id=3Q4BIBCFEYXICWG3CEFEUIBAMBNZWJ7FNCMMB4DHJ73RHKVXGBKQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.288897151 (106h2m21.719977283s) [noise=true , address=104.36.82.58:4298 , piece_id=I76VJFWH36OITPI2LEHTUPWFSC566LKQZBGM7VVJ6744TYN2QESA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.298496038 (106h2m21.729576169s) [noise=true , address=104.36.82.36:4641 , piece_id=3A5FI2NFX2Y2QYX6JMEMIPXAYU6ZGB662F6PZGGY6W456MNN4H2Q, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.305049777 (106h2m21.736129908s) [noise=true , address=104.36.82.35:4585 , piece_id=EA7XTERIGFLJMV33KWPJJEFXKDNYHSVJIBYES33AANWX6RFO3Z4A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.319397510 (106h2m21.750477631s) [noise=true , address=104.36.82.21:4488 , piece_id=ZJC77OAGWGPQHJLXGSY2JUS4POTFI4UYW2ORN2OET4U5QCCK2RYA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.324171042 (106h2m21.755251173s) [noise=true , address=104.36.82.8:4988 , piece_id=KZO4HIEGBYBTXIUMP7YETN5WBBNS2ZMGOA3W6H377RF5RV3RIGRQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.325044093 (106h2m21.756124224s) [noise=true , address=104.36.82.39:4758 , piece_id=6GQJIBOJ6XK6RILWWEPLNQH7OPKSYWFMQG3SXYTVONZRLV3YDU7A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.326189485 (106h2m21.757269616s) [noise=true , address=104.36.82.4:4833 , piece_id=WOJB675NZW5MONNZ7MHECJEYFRFUQDDQFCG4EATF2DTTQOY5FPIA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.358731589 (106h2m21.78981172s) [noise=true , address=104.36.82.15:4267 , piece_id=YOBGDLVCGO7Z6XLQQ6K7INK36KDGOHZNA6L2ROJF2VVC3YOE5CEQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.364728292 (106h2m21.795808413s) [noise=true , address=104.36.82.37:4686 , piece_id=TWWJUP6LY7VX4BW7PSTTZK53JTGSVBP2Y6NQCR3PXT44WCLGY2WA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.386279758 (106h2m21.817359889s) [noise=true , address=77.90.31.21:4906 , piece_id=FLIZ63F7NOFHDUEDYOAGMMGEGS473UQPXY2C3WKX67MGWJJEZBIA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.393790007 (106h2m21.824870158s) [noise=true , address=104.36.82.44:4876 , piece_id=576P5WMAQ6BCLTG4UAJX2ZCHGAMKCD47Z6YQBWJKHI5MQGGXT4XA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.393863398 (106h2m21.824943529s) [seg_pos={0 0}, upload=110366]: Segment reached optimal threshold of 14 pieces. Aug 13 03:28:27.393993957 (106h2m21.825074088s) [noise=true , address=104.36.82.46:4496 , piece_id=K6J6AEVYWM63WM6IFDXSGBUWK6WX72R26RLVLRZ64VSF4H2744JA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394051778 (106h2m21.82513191s) [noise=true , address=104.36.82.5:4866 , piece_id=22IXIQNEVDD6VILGPIQMS46JFYVQDI3BNI633KABQLCDBQQB22ZA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zUcYfJE8xRW2Tow7bGB3PeNrVBUzVWR82JADkVwPdSr4CTPyr): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394083198 (106h2m21.82516333s) [noise=true , address=104.36.82.60:4363 , piece_id=SRLXSTGJIEBAM4H6UE6AF32SJY7N2ZHJPPDOXKTUN44ZI3LT2I5A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7): protocol: expected piece hash; write tcp 188.166.225.170:42920->104.36.82.60:4363: use of closed network connection; write tcp 188.166.225.170:42920->104.36.82.60:4363: use of closed network connection; EOF Aug 13 03:28:27.394101453 (106h2m21.825181574s) [noise=true , address=79.127.220.98:4422 , piece_id=R6WFYRTY4IEQ4ORVBDWP5FBP676QRCJDPKSBNJDXL64C2OH2SWDA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MScXvmPpGZ6HRRwaoBMrZR3TvhDQh4nZkpSYggRQGzP1JbyVE): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394138204 (106h2m21.825218335s) [seg_pos={0 0}, upload=110366]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 13 03:28:27.394212385 (106h2m21.825292516s) [upload=110366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 13 03:28:27.394217095 (106h2m21.825297226s) [upload=110366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:28:27.720702752 (106h2m22.151782883s) [upload=110366]: Done uploading segments... Aug 13 03:28:27.720720255 (106h2m22.151800386s) [upload=110366]: Upload finished. err: <nil> Duration: 5.757883389s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.099504963s FAIL (2025-08-13 03:30:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 13 03:23:34.987469695 (105h57m29.418549826s) [upload=110284]: Starting upload Aug 13 03:23:34.987520002 (105h57m29.418600133s) [upload=110284]: Done starting upload Aug 13 03:23:34.987570067 (105h57m29.418650198s) [upload=110284]: Uploading segments... Aug 13 03:23:34.987604243 (105h57m29.418684374s) [upload=110284]: Got next segment. Inline: true Aug 13 03:23:34.987608641 (105h57m29.418688772s) [upload=110284]: Waiting for error group managing segments... Aug 13 03:23:34.987640973 (105h57m29.418721094s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:23:34.987644710 (105h57m29.418724841s) [upload=110284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:23:34.987647044 (105h57m29.418727165s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:23:35.199946062 (105h57m29.631026193s) [upload=110284]: Done uploading segments... Aug 13 03:23:35.199964076 (105h57m29.631044207s) [upload=110284]: Upload finished. err: metaclient: Too Many Requests Duration:* 2.099543758s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 1.821110556s FAIL (2025-08-13 03:30:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 13 03:06:41.898033585 (105h40m36.329113716s) [upload=110007]: Starting upload Aug 13 03:06:41.898093971 (105h40m36.329174102s) [upload=110007]: Done starting upload Aug 13 03:06:41.898140650 (105h40m36.329220781s) [upload=110007]: Uploading segments... Aug 13 03:06:41.898172331 (105h40m36.329252462s) [upload=110007]: Got next segment. Inline: true Aug 13 03:06:41.898176078 (105h40m36.329256209s) [upload=110007]: Waiting for error group managing segments... Aug 13 03:06:41.898208831 (105h40m36.329288962s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:06:41.898212778 (105h40m36.329292909s) [upload=110007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:06:41.898215563 (105h40m36.329295694s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:07:11.244546166 (105h41m5.675626307s) [upload=110007]: Done uploading segments... Aug 13 03:07:11.244585330 (105h41m5.675665482s) [upload=110007]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.821143719s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X.......X...........X...............X...................X..] duration: 5.819491807s FAIL (2025-08-13 03:30:22 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 13 03:28:24.870384288 (106h2m19.301464419s) [upload=110366]: Starting upload Aug 13 03:28:24.870426058 (106h2m19.30150618s) [upload=110366]: Done starting upload Aug 13 03:28:24.870488848 (106h2m19.301568979s) [upload=110366]: Uploading segments... Aug 13 03:28:24.870604439 (106h2m19.30168457s) [upload=110366]: Got next segment. Inline: false Aug 13 03:28:24.870637743 (106h2m19.301717874s) [upload=110366]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:28:24.870642031 (106h2m19.301722162s) [upload=110366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 13 03:28:25.321382698 (106h2m19.752462879s) [seg_pos={0 0}, upload=110366]: Begin upload segment... Aug 13 03:28:25.321441521 (106h2m19.752521652s) [seg_pos={0 0}, upload=110366]: Done begin upload segment. Aug 13 03:28:25.321465406 (106h2m19.752545527s) [noise=true , address=104.36.82.44:4876 , piece_id=576P5WMAQ6BCLTG4UAJX2ZCHGAMKCD47Z6YQBWJKHI5MQGGXT4XA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.321492969 (106h2m19.75257309s) [noise=true , address=104.36.82.37:4686 , piece_id=TWWJUP6LY7VX4BW7PSTTZK53JTGSVBP2Y6NQCR3PXT44WCLGY2WA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.321866553 (106h2m19.752946694s) [noise=true , address=104.36.82.4:4833 , piece_id=WOJB675NZW5MONNZ7MHECJEYFRFUQDDQFCG4EATF2DTTQOY5FPIA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322159584 (106h2m19.753239715s) [noise=true , address=104.36.82.38:4716 , piece_id=3Q4BIBCFEYXICWG3CEFEUIBAMBNZWJ7FNCMMB4DHJ73RHKVXGBKQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322826730 (106h2m19.753906862s) [noise=true , address=104.36.82.35:4585 , piece_id=EA7XTERIGFLJMV33KWPJJEFXKDNYHSVJIBYES33AANWX6RFO3Z4A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322949396 (106h2m19.754029517s) [noise=true , address=104.36.82.15:4267 , piece_id=YOBGDLVCGO7Z6XLQQ6K7INK36KDGOHZNA6L2ROJF2VVC3YOE5CEQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323182792 (106h2m19.754262923s) [noise=true , address=64.49.13.139:4473 , piece_id=GYBREOGJD222LYFTSGTYHYLM4T22ZKSQYKN4GNW42ZQSNZQOHQJA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323412270 (106h2m19.754492402s) [noise=true , address=104.36.82.60:4363 , piece_id=SRLXSTGJIEBAM4H6UE6AF32SJY7N2ZHJPPDOXKTUN44ZI3LT2I5A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323740879 (106h2m19.75482101s) [noise=true , address=104.36.82.5:4866 , piece_id=22IXIQNEVDD6VILGPIQMS46JFYVQDI3BNI633KABQLCDBQQB22ZA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.324341969 (106h2m19.755422101s) [noise=true , address=104.36.82.46:4496 , piece_id=K6J6AEVYWM63WM6IFDXSGBUWK6WX72R26RLVLRZ64VSF4H2744JA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.324756232 (106h2m19.755836363s) [noise=true , address=104.36.82.8:4988 , piece_id=KZO4HIEGBYBTXIUMP7YETN5WBBNS2ZMGOA3W6H377RF5RV3RIGRQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325088147 (106h2m19.756168278s) [noise=true , address=64.49.13.139:4478 , piece_id=ZLJZ3QKNWE7LHFZSZEFOSZ6EN27KIAJFEEY7TVL4YW2VZXQM6IOA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325336562 (106h2m19.756416693s) [noise=true , address=104.36.82.36:4641 , piece_id=3A5FI2NFX2Y2QYX6JMEMIPXAYU6ZGB662F6PZGGY6W456MNN4H2Q, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325678096 (106h2m19.756758217s) [noise=true , address=104.36.82.39:4758 , piece_id=6GQJIBOJ6XK6RILWWEPLNQH7OPKSYWFMQG3SXYTVONZRLV3YDU7A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325997056 (106h2m19.757077187s) [noise=true , address=79.127.220.98:4422 , piece_id=R6WFYRTY4IEQ4ORVBDWP5FBP676QRCJDPKSBNJDXL64C2OH2SWDA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326218289 (106h2m19.75729842s) [noise=true , address=104.36.82.21:4488 , piece_id=ZJC77OAGWGPQHJLXGSY2JUS4POTFI4UYW2ORN2OET4U5QCCK2RYA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326380920 (106h2m19.757461051s) [noise=true , address=77.90.31.21:4906 , piece_id=FLIZ63F7NOFHDUEDYOAGMMGEGS473UQPXY2C3WKX67MGWJJEZBIA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326717245 (106h2m19.757797376s) [noise=true , address=104.36.82.58:4298 , piece_id=I76VJFWH36OITPI2LEHTUPWFSC566LKQZBGM7VVJ6744TYN2QESA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:26.509048537 (106h2m20.940128668s) [upload=110366]: Next returned nil segment Aug 13 03:28:26.509066711 (106h2m20.940146842s) [upload=110366]: Waiting for error group managing segments... Aug 13 03:28:26.809943799 (106h2m21.24102393s) [noise=true , address=64.49.13.139:4478 , piece_id=ZLJZ3QKNWE7LHFZSZEFOSZ6EN27KIAJFEEY7TVL4YW2VZXQM6IOA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:26.884546048 (106h2m21.315626179s) [noise=true , address=64.49.13.139:4473 , piece_id=GYBREOGJD222LYFTSGTYHYLM4T22ZKSQYKN4GNW42ZQSNZQOHQJA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.268324420 (106h2m21.699404611s) [noise=true , address=104.36.82.38:4716 , piece_id=3Q4BIBCFEYXICWG3CEFEUIBAMBNZWJ7FNCMMB4DHJ73RHKVXGBKQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.288897151 (106h2m21.719977283s) [noise=true , address=104.36.82.58:4298 , piece_id=I76VJFWH36OITPI2LEHTUPWFSC566LKQZBGM7VVJ6744TYN2QESA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.298496038 (106h2m21.729576169s) [noise=true , address=104.36.82.36:4641 , piece_id=3A5FI2NFX2Y2QYX6JMEMIPXAYU6ZGB662F6PZGGY6W456MNN4H2Q, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.305049777 (106h2m21.736129908s) [noise=true , address=104.36.82.35:4585 , piece_id=EA7XTERIGFLJMV33KWPJJEFXKDNYHSVJIBYES33AANWX6RFO3Z4A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.319397510 (106h2m21.750477631s) [noise=true , address=104.36.82.21:4488 , piece_id=ZJC77OAGWGPQHJLXGSY2JUS4POTFI4UYW2ORN2OET4U5QCCK2RYA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.324171042 (106h2m21.755251173s) [noise=true , address=104.36.82.8:4988 , piece_id=KZO4HIEGBYBTXIUMP7YETN5WBBNS2ZMGOA3W6H377RF5RV3RIGRQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.325044093 (106h2m21.756124224s) [noise=true , address=104.36.82.39:4758 , piece_id=6GQJIBOJ6XK6RILWWEPLNQH7OPKSYWFMQG3SXYTVONZRLV3YDU7A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.326189485 (106h2m21.757269616s) [noise=true , address=104.36.82.4:4833 , piece_id=WOJB675NZW5MONNZ7MHECJEYFRFUQDDQFCG4EATF2DTTQOY5FPIA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.358731589 (106h2m21.78981172s) [noise=true , address=104.36.82.15:4267 , piece_id=YOBGDLVCGO7Z6XLQQ6K7INK36KDGOHZNA6L2ROJF2VVC3YOE5CEQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.364728292 (106h2m21.795808413s) [noise=true , address=104.36.82.37:4686 , piece_id=TWWJUP6LY7VX4BW7PSTTZK53JTGSVBP2Y6NQCR3PXT44WCLGY2WA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.386279758 (106h2m21.817359889s) [noise=true , address=77.90.31.21:4906 , piece_id=FLIZ63F7NOFHDUEDYOAGMMGEGS473UQPXY2C3WKX67MGWJJEZBIA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.393790007 (106h2m21.824870158s) [noise=true , address=104.36.82.44:4876 , piece_id=576P5WMAQ6BCLTG4UAJX2ZCHGAMKCD47Z6YQBWJKHI5MQGGXT4XA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.393863398 (106h2m21.824943529s) [seg_pos={0 0}, upload=110366]: Segment reached optimal threshold of 14 pieces. Aug 13 03:28:27.393993957 (106h2m21.825074088s) [noise=true , address=104.36.82.46:4496 , piece_id=K6J6AEVYWM63WM6IFDXSGBUWK6WX72R26RLVLRZ64VSF4H2744JA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394051778 (106h2m21.82513191s) [noise=true , address=104.36.82.5:4866 , piece_id=22IXIQNEVDD6VILGPIQMS46JFYVQDI3BNI633KABQLCDBQQB22ZA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zUcYfJE8xRW2Tow7bGB3PeNrVBUzVWR82JADkVwPdSr4CTPyr): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394083198 (106h2m21.82516333s) [noise=true , address=104.36.82.60:4363 , piece_id=SRLXSTGJIEBAM4H6UE6AF32SJY7N2ZHJPPDOXKTUN44ZI3LT2I5A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7): protocol: expected piece hash; write tcp 188.166.225.170:42920->104.36.82.60:4363: use of closed network connection; write tcp 188.166.225.170:42920->104.36.82.60:4363: use of closed network connection; EOF Aug 13 03:28:27.394101453 (106h2m21.825181574s) [noise=true , address=79.127.220.98:4422 , piece_id=R6WFYRTY4IEQ4ORVBDWP5FBP676QRCJDPKSBNJDXL64C2OH2SWDA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MScXvmPpGZ6HRRwaoBMrZR3TvhDQh4nZkpSYggRQGzP1JbyVE): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394138204 (106h2m21.825218335s) [seg_pos={0 0}, upload=110366]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 13 03:28:27.394212385 (106h2m21.825292516s) [upload=110366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 13 03:28:27.394217095 (106h2m21.825297226s) [upload=110366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:28:27.720702752 (106h2m22.151782883s) [upload=110366]: Done uploading segments... Aug 13 03:28:27.720720255 (106h2m22.151800386s) [upload=110366]: Upload finished. err: <nil> Duration: 5.819523118s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................................X.....XX.............] duration: 7.235601357s FAIL (2025-08-13 03:30:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.235634461s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 1.859193817s FAIL (2025-08-13 03:30:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 13 03:23:34.987469695 (105h57m29.418549826s) [upload=110284]: Starting upload Aug 13 03:23:34.987520002 (105h57m29.418600133s) [upload=110284]: Done starting upload Aug 13 03:23:34.987570067 (105h57m29.418650198s) [upload=110284]: Uploading segments... Aug 13 03:23:34.987604243 (105h57m29.418684374s) [upload=110284]: Got next segment. Inline: true Aug 13 03:23:34.987608641 (105h57m29.418688772s) [upload=110284]: Waiting for error group managing segments... Aug 13 03:23:34.987640973 (105h57m29.418721094s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:23:34.987644710 (105h57m29.418724841s) [upload=110284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:23:34.987647044 (105h57m29.418727165s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:23:35.199946062 (105h57m29.631026193s) [upload=110284]: Done uploading segments... Aug 13 03:23:35.199964076 (105h57m29.631044207s) [upload=110284]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.859230207s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 1.871016158s FAIL (2025-08-13 03:29:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 13 03:06:41.898033585 (105h40m36.329113716s) [upload=110007]: Starting upload Aug 13 03:06:41.898093971 (105h40m36.329174102s) [upload=110007]: Done starting upload Aug 13 03:06:41.898140650 (105h40m36.329220781s) [upload=110007]: Uploading segments... Aug 13 03:06:41.898172331 (105h40m36.329252462s) [upload=110007]: Got next segment. Inline: true Aug 13 03:06:41.898176078 (105h40m36.329256209s) [upload=110007]: Waiting for error group managing segments... Aug 13 03:06:41.898208831 (105h40m36.329288962s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:06:41.898212778 (105h40m36.329292909s) [upload=110007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:06:41.898215563 (105h40m36.329295694s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:07:11.244546166 (105h41m5.675626307s) [upload=110007]: Done uploading segments... Aug 13 03:07:11.244585330 (105h41m5.675665482s) [upload=110007]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.871044392s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.005811566s FAIL (2025-08-13 03:29:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 13 03:23:34.987469695 (105h57m29.418549826s) [upload=110284]: Starting upload Aug 13 03:23:34.987520002 (105h57m29.418600133s) [upload=110284]: Done starting upload Aug 13 03:23:34.987570067 (105h57m29.418650198s) [upload=110284]: Uploading segments... Aug 13 03:23:34.987604243 (105h57m29.418684374s) [upload=110284]: Got next segment. Inline: true Aug 13 03:23:34.987608641 (105h57m29.418688772s) [upload=110284]: Waiting for error group managing segments... Aug 13 03:23:34.987640973 (105h57m29.418721094s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:23:34.987644710 (105h57m29.418724841s) [upload=110284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:23:34.987647044 (105h57m29.418727165s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:23:35.199946062 (105h57m29.631026193s) [upload=110284]: Done uploading segments... Aug 13 03:23:35.199964076 (105h57m29.631044207s) [upload=110284]: Upload finished. err: metaclient: Too Many Requests Duration:* 2.00585617s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X.......X...........X...............X...................X.] duration: 5.604925692s FAIL (2025-08-13 03:29:29 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 13 03:28:24.870384288 (106h2m19.301464419s) [upload=110366]: Starting upload Aug 13 03:28:24.870426058 (106h2m19.30150618s) [upload=110366]: Done starting upload Aug 13 03:28:24.870488848 (106h2m19.301568979s) [upload=110366]: Uploading segments... Aug 13 03:28:24.870604439 (106h2m19.30168457s) [upload=110366]: Got next segment. Inline: false Aug 13 03:28:24.870637743 (106h2m19.301717874s) [upload=110366]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:28:24.870642031 (106h2m19.301722162s) [upload=110366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 13 03:28:25.321382698 (106h2m19.752462879s) [seg_pos={0 0}, upload=110366]: Begin upload segment... Aug 13 03:28:25.321441521 (106h2m19.752521652s) [seg_pos={0 0}, upload=110366]: Done begin upload segment. Aug 13 03:28:25.321465406 (106h2m19.752545527s) [noise=true , address=104.36.82.44:4876 , piece_id=576P5WMAQ6BCLTG4UAJX2ZCHGAMKCD47Z6YQBWJKHI5MQGGXT4XA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.321492969 (106h2m19.75257309s) [noise=true , address=104.36.82.37:4686 , piece_id=TWWJUP6LY7VX4BW7PSTTZK53JTGSVBP2Y6NQCR3PXT44WCLGY2WA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.321866553 (106h2m19.752946694s) [noise=true , address=104.36.82.4:4833 , piece_id=WOJB675NZW5MONNZ7MHECJEYFRFUQDDQFCG4EATF2DTTQOY5FPIA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322159584 (106h2m19.753239715s) [noise=true , address=104.36.82.38:4716 , piece_id=3Q4BIBCFEYXICWG3CEFEUIBAMBNZWJ7FNCMMB4DHJ73RHKVXGBKQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322826730 (106h2m19.753906862s) [noise=true , address=104.36.82.35:4585 , piece_id=EA7XTERIGFLJMV33KWPJJEFXKDNYHSVJIBYES33AANWX6RFO3Z4A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322949396 (106h2m19.754029517s) [noise=true , address=104.36.82.15:4267 , piece_id=YOBGDLVCGO7Z6XLQQ6K7INK36KDGOHZNA6L2ROJF2VVC3YOE5CEQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323182792 (106h2m19.754262923s) [noise=true , address=64.49.13.139:4473 , piece_id=GYBREOGJD222LYFTSGTYHYLM4T22ZKSQYKN4GNW42ZQSNZQOHQJA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323412270 (106h2m19.754492402s) [noise=true , address=104.36.82.60:4363 , piece_id=SRLXSTGJIEBAM4H6UE6AF32SJY7N2ZHJPPDOXKTUN44ZI3LT2I5A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323740879 (106h2m19.75482101s) [noise=true , address=104.36.82.5:4866 , piece_id=22IXIQNEVDD6VILGPIQMS46JFYVQDI3BNI633KABQLCDBQQB22ZA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.324341969 (106h2m19.755422101s) [noise=true , address=104.36.82.46:4496 , piece_id=K6J6AEVYWM63WM6IFDXSGBUWK6WX72R26RLVLRZ64VSF4H2744JA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.324756232 (106h2m19.755836363s) [noise=true , address=104.36.82.8:4988 , piece_id=KZO4HIEGBYBTXIUMP7YETN5WBBNS2ZMGOA3W6H377RF5RV3RIGRQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325088147 (106h2m19.756168278s) [noise=true , address=64.49.13.139:4478 , piece_id=ZLJZ3QKNWE7LHFZSZEFOSZ6EN27KIAJFEEY7TVL4YW2VZXQM6IOA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325336562 (106h2m19.756416693s) [noise=true , address=104.36.82.36:4641 , piece_id=3A5FI2NFX2Y2QYX6JMEMIPXAYU6ZGB662F6PZGGY6W456MNN4H2Q, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325678096 (106h2m19.756758217s) [noise=true , address=104.36.82.39:4758 , piece_id=6GQJIBOJ6XK6RILWWEPLNQH7OPKSYWFMQG3SXYTVONZRLV3YDU7A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325997056 (106h2m19.757077187s) [noise=true , address=79.127.220.98:4422 , piece_id=R6WFYRTY4IEQ4ORVBDWP5FBP676QRCJDPKSBNJDXL64C2OH2SWDA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326218289 (106h2m19.75729842s) [noise=true , address=104.36.82.21:4488 , piece_id=ZJC77OAGWGPQHJLXGSY2JUS4POTFI4UYW2ORN2OET4U5QCCK2RYA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326380920 (106h2m19.757461051s) [noise=true , address=77.90.31.21:4906 , piece_id=FLIZ63F7NOFHDUEDYOAGMMGEGS473UQPXY2C3WKX67MGWJJEZBIA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326717245 (106h2m19.757797376s) [noise=true , address=104.36.82.58:4298 , piece_id=I76VJFWH36OITPI2LEHTUPWFSC566LKQZBGM7VVJ6744TYN2QESA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:26.509048537 (106h2m20.940128668s) [upload=110366]: Next returned nil segment Aug 13 03:28:26.509066711 (106h2m20.940146842s) [upload=110366]: Waiting for error group managing segments... Aug 13 03:28:26.809943799 (106h2m21.24102393s) [noise=true , address=64.49.13.139:4478 , piece_id=ZLJZ3QKNWE7LHFZSZEFOSZ6EN27KIAJFEEY7TVL4YW2VZXQM6IOA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:26.884546048 (106h2m21.315626179s) [noise=true , address=64.49.13.139:4473 , piece_id=GYBREOGJD222LYFTSGTYHYLM4T22ZKSQYKN4GNW42ZQSNZQOHQJA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.268324420 (106h2m21.699404611s) [noise=true , address=104.36.82.38:4716 , piece_id=3Q4BIBCFEYXICWG3CEFEUIBAMBNZWJ7FNCMMB4DHJ73RHKVXGBKQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.288897151 (106h2m21.719977283s) [noise=true , address=104.36.82.58:4298 , piece_id=I76VJFWH36OITPI2LEHTUPWFSC566LKQZBGM7VVJ6744TYN2QESA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.298496038 (106h2m21.729576169s) [noise=true , address=104.36.82.36:4641 , piece_id=3A5FI2NFX2Y2QYX6JMEMIPXAYU6ZGB662F6PZGGY6W456MNN4H2Q, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.305049777 (106h2m21.736129908s) [noise=true , address=104.36.82.35:4585 , piece_id=EA7XTERIGFLJMV33KWPJJEFXKDNYHSVJIBYES33AANWX6RFO3Z4A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.319397510 (106h2m21.750477631s) [noise=true , address=104.36.82.21:4488 , piece_id=ZJC77OAGWGPQHJLXGSY2JUS4POTFI4UYW2ORN2OET4U5QCCK2RYA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.324171042 (106h2m21.755251173s) [noise=true , address=104.36.82.8:4988 , piece_id=KZO4HIEGBYBTXIUMP7YETN5WBBNS2ZMGOA3W6H377RF5RV3RIGRQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.325044093 (106h2m21.756124224s) [noise=true , address=104.36.82.39:4758 , piece_id=6GQJIBOJ6XK6RILWWEPLNQH7OPKSYWFMQG3SXYTVONZRLV3YDU7A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.326189485 (106h2m21.757269616s) [noise=true , address=104.36.82.4:4833 , piece_id=WOJB675NZW5MONNZ7MHECJEYFRFUQDDQFCG4EATF2DTTQOY5FPIA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.358731589 (106h2m21.78981172s) [noise=true , address=104.36.82.15:4267 , piece_id=YOBGDLVCGO7Z6XLQQ6K7INK36KDGOHZNA6L2ROJF2VVC3YOE5CEQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.364728292 (106h2m21.795808413s) [noise=true , address=104.36.82.37:4686 , piece_id=TWWJUP6LY7VX4BW7PSTTZK53JTGSVBP2Y6NQCR3PXT44WCLGY2WA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.386279758 (106h2m21.817359889s) [noise=true , address=77.90.31.21:4906 , piece_id=FLIZ63F7NOFHDUEDYOAGMMGEGS473UQPXY2C3WKX67MGWJJEZBIA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.393790007 (106h2m21.824870158s) [noise=true , address=104.36.82.44:4876 , piece_id=576P5WMAQ6BCLTG4UAJX2ZCHGAMKCD47Z6YQBWJKHI5MQGGXT4XA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.393863398 (106h2m21.824943529s) [seg_pos={0 0}, upload=110366]: Segment reached optimal threshold of 14 pieces. Aug 13 03:28:27.393993957 (106h2m21.825074088s) [noise=true , address=104.36.82.46:4496 , piece_id=K6J6AEVYWM63WM6IFDXSGBUWK6WX72R26RLVLRZ64VSF4H2744JA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394051778 (106h2m21.82513191s) [noise=true , address=104.36.82.5:4866 , piece_id=22IXIQNEVDD6VILGPIQMS46JFYVQDI3BNI633KABQLCDBQQB22ZA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zUcYfJE8xRW2Tow7bGB3PeNrVBUzVWR82JADkVwPdSr4CTPyr): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394083198 (106h2m21.82516333s) [noise=true , address=104.36.82.60:4363 , piece_id=SRLXSTGJIEBAM4H6UE6AF32SJY7N2ZHJPPDOXKTUN44ZI3LT2I5A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7): protocol: expected piece hash; write tcp 188.166.225.170:42920->104.36.82.60:4363: use of closed network connection; write tcp 188.166.225.170:42920->104.36.82.60:4363: use of closed network connection; EOF Aug 13 03:28:27.394101453 (106h2m21.825181574s) [noise=true , address=79.127.220.98:4422 , piece_id=R6WFYRTY4IEQ4ORVBDWP5FBP676QRCJDPKSBNJDXL64C2OH2SWDA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MScXvmPpGZ6HRRwaoBMrZR3TvhDQh4nZkpSYggRQGzP1JbyVE): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394138204 (106h2m21.825218335s) [seg_pos={0 0}, upload=110366]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 13 03:28:27.394212385 (106h2m21.825292516s) [upload=110366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 13 03:28:27.394217095 (106h2m21.825297226s) [upload=110366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:28:27.720702752 (106h2m22.151782883s) [upload=110366]: Done uploading segments... Aug 13 03:28:27.720720255 (106h2m22.151800386s) [upload=110366]: Upload finished. err: <nil> Duration: 5.604951562s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 1.887571494s FAIL (2025-08-13 03:29:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 13 03:06:41.898033585 (105h40m36.329113716s) [upload=110007]: Starting upload Aug 13 03:06:41.898093971 (105h40m36.329174102s) [upload=110007]: Done starting upload Aug 13 03:06:41.898140650 (105h40m36.329220781s) [upload=110007]: Uploading segments... Aug 13 03:06:41.898172331 (105h40m36.329252462s) [upload=110007]: Got next segment. Inline: true Aug 13 03:06:41.898176078 (105h40m36.329256209s) [upload=110007]: Waiting for error group managing segments... Aug 13 03:06:41.898208831 (105h40m36.329288962s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:06:41.898212778 (105h40m36.329292909s) [upload=110007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:06:41.898215563 (105h40m36.329295694s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:07:11.244546166 (105h41m5.675626307s) [upload=110007]: Done uploading segments... Aug 13 03:07:11.244585330 (105h41m5.675665482s) [upload=110007]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.887620739s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................X.....XX............] duration: 10.267060519s FAIL (2025-08-13 03:29:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.267080937s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................X.....XX...........] duration: 7.767569653s FAIL (2025-08-13 03:28:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.767593647s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 1.870276674s FAIL (2025-08-13 03:28:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 13 03:06:41.898033585 (105h40m36.329113716s) [upload=110007]: Starting upload Aug 13 03:06:41.898093971 (105h40m36.329174102s) [upload=110007]: Done starting upload Aug 13 03:06:41.898140650 (105h40m36.329220781s) [upload=110007]: Uploading segments... Aug 13 03:06:41.898172331 (105h40m36.329252462s) [upload=110007]: Got next segment. Inline: true Aug 13 03:06:41.898176078 (105h40m36.329256209s) [upload=110007]: Waiting for error group managing segments... Aug 13 03:06:41.898208831 (105h40m36.329288962s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:06:41.898212778 (105h40m36.329292909s) [upload=110007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:06:41.898215563 (105h40m36.329295694s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:07:11.244546166 (105h41m5.675626307s) [upload=110007]: Done uploading segments... Aug 13 03:07:11.244585330 (105h41m5.675665482s) [upload=110007]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.87032142s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.011234454s FAIL (2025-08-13 03:28:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 13 03:23:34.987469695 (105h57m29.418549826s) [upload=110284]: Starting upload Aug 13 03:23:34.987520002 (105h57m29.418600133s) [upload=110284]: Done starting upload Aug 13 03:23:34.987570067 (105h57m29.418650198s) [upload=110284]: Uploading segments... Aug 13 03:23:34.987604243 (105h57m29.418684374s) [upload=110284]: Got next segment. Inline: true Aug 13 03:23:34.987608641 (105h57m29.418688772s) [upload=110284]: Waiting for error group managing segments... Aug 13 03:23:34.987640973 (105h57m29.418721094s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:23:34.987644710 (105h57m29.418724841s) [upload=110284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:23:34.987647044 (105h57m29.418727165s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:23:35.199946062 (105h57m29.631026193s) [upload=110284]: Done uploading segments... Aug 13 03:23:35.199964076 (105h57m29.631044207s) [upload=110284]: Upload finished. err: metaclient: Too Many Requests Duration:* 2.011256094s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [X..X.......X...........X...............X...................X] duration: 3.987565073s FAIL (2025-08-13 03:28:28 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== Aug 13 03:28:24.870384288 (106h2m19.301464419s) [upload=110366]: Starting upload Aug 13 03:28:24.870426058 (106h2m19.30150618s) [upload=110366]: Done starting upload Aug 13 03:28:24.870488848 (106h2m19.301568979s) [upload=110366]: Uploading segments... Aug 13 03:28:24.870604439 (106h2m19.30168457s) [upload=110366]: Got next segment. Inline: false Aug 13 03:28:24.870637743 (106h2m19.301717874s) [upload=110366]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:28:24.870642031 (106h2m19.301722162s) [upload=110366]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 13 03:28:25.321382698 (106h2m19.752462879s) [seg_pos={0 0}, upload=110366]: Begin upload segment... Aug 13 03:28:25.321441521 (106h2m19.752521652s) [seg_pos={0 0}, upload=110366]: Done begin upload segment. Aug 13 03:28:25.321465406 (106h2m19.752545527s) [noise=true , address=104.36.82.44:4876 , piece_id=576P5WMAQ6BCLTG4UAJX2ZCHGAMKCD47Z6YQBWJKHI5MQGGXT4XA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.321492969 (106h2m19.75257309s) [noise=true , address=104.36.82.37:4686 , piece_id=TWWJUP6LY7VX4BW7PSTTZK53JTGSVBP2Y6NQCR3PXT44WCLGY2WA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.321866553 (106h2m19.752946694s) [noise=true , address=104.36.82.4:4833 , piece_id=WOJB675NZW5MONNZ7MHECJEYFRFUQDDQFCG4EATF2DTTQOY5FPIA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322159584 (106h2m19.753239715s) [noise=true , address=104.36.82.38:4716 , piece_id=3Q4BIBCFEYXICWG3CEFEUIBAMBNZWJ7FNCMMB4DHJ73RHKVXGBKQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322826730 (106h2m19.753906862s) [noise=true , address=104.36.82.35:4585 , piece_id=EA7XTERIGFLJMV33KWPJJEFXKDNYHSVJIBYES33AANWX6RFO3Z4A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.322949396 (106h2m19.754029517s) [noise=true , address=104.36.82.15:4267 , piece_id=YOBGDLVCGO7Z6XLQQ6K7INK36KDGOHZNA6L2ROJF2VVC3YOE5CEQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323182792 (106h2m19.754262923s) [noise=true , address=64.49.13.139:4473 , piece_id=GYBREOGJD222LYFTSGTYHYLM4T22ZKSQYKN4GNW42ZQSNZQOHQJA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323412270 (106h2m19.754492402s) [noise=true , address=104.36.82.60:4363 , piece_id=SRLXSTGJIEBAM4H6UE6AF32SJY7N2ZHJPPDOXKTUN44ZI3LT2I5A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.323740879 (106h2m19.75482101s) [noise=true , address=104.36.82.5:4866 , piece_id=22IXIQNEVDD6VILGPIQMS46JFYVQDI3BNI633KABQLCDBQQB22ZA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.324341969 (106h2m19.755422101s) [noise=true , address=104.36.82.46:4496 , piece_id=K6J6AEVYWM63WM6IFDXSGBUWK6WX72R26RLVLRZ64VSF4H2744JA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.324756232 (106h2m19.755836363s) [noise=true , address=104.36.82.8:4988 , piece_id=KZO4HIEGBYBTXIUMP7YETN5WBBNS2ZMGOA3W6H377RF5RV3RIGRQ, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325088147 (106h2m19.756168278s) [noise=true , address=64.49.13.139:4478 , piece_id=ZLJZ3QKNWE7LHFZSZEFOSZ6EN27KIAJFEEY7TVL4YW2VZXQM6IOA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325336562 (106h2m19.756416693s) [noise=true , address=104.36.82.36:4641 , piece_id=3A5FI2NFX2Y2QYX6JMEMIPXAYU6ZGB662F6PZGGY6W456MNN4H2Q, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325678096 (106h2m19.756758217s) [noise=true , address=104.36.82.39:4758 , piece_id=6GQJIBOJ6XK6RILWWEPLNQH7OPKSYWFMQG3SXYTVONZRLV3YDU7A, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.325997056 (106h2m19.757077187s) [noise=true , address=79.127.220.98:4422 , piece_id=R6WFYRTY4IEQ4ORVBDWP5FBP676QRCJDPKSBNJDXL64C2OH2SWDA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326218289 (106h2m19.75729842s) [noise=true , address=104.36.82.21:4488 , piece_id=ZJC77OAGWGPQHJLXGSY2JUS4POTFI4UYW2ORN2OET4U5QCCK2RYA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326380920 (106h2m19.757461051s) [noise=true , address=77.90.31.21:4906 , piece_id=FLIZ63F7NOFHDUEDYOAGMMGEGS473UQPXY2C3WKX67MGWJJEZBIA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:25.326717245 (106h2m19.757797376s) [noise=true , address=104.36.82.58:4298 , piece_id=I76VJFWH36OITPI2LEHTUPWFSC566LKQZBGM7VVJ6744TYN2QESA, seg_pos={0 0}, upload=110366]: Uploading piece... Aug 13 03:28:26.509048537 (106h2m20.940128668s) [upload=110366]: Next returned nil segment Aug 13 03:28:26.509066711 (106h2m20.940146842s) [upload=110366]: Waiting for error group managing segments... Aug 13 03:28:26.809943799 (106h2m21.24102393s) [noise=true , address=64.49.13.139:4478 , piece_id=ZLJZ3QKNWE7LHFZSZEFOSZ6EN27KIAJFEEY7TVL4YW2VZXQM6IOA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:26.884546048 (106h2m21.315626179s) [noise=true , address=64.49.13.139:4473 , piece_id=GYBREOGJD222LYFTSGTYHYLM4T22ZKSQYKN4GNW42ZQSNZQOHQJA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.268324420 (106h2m21.699404611s) [noise=true , address=104.36.82.38:4716 , piece_id=3Q4BIBCFEYXICWG3CEFEUIBAMBNZWJ7FNCMMB4DHJ73RHKVXGBKQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.288897151 (106h2m21.719977283s) [noise=true , address=104.36.82.58:4298 , piece_id=I76VJFWH36OITPI2LEHTUPWFSC566LKQZBGM7VVJ6744TYN2QESA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.298496038 (106h2m21.729576169s) [noise=true , address=104.36.82.36:4641 , piece_id=3A5FI2NFX2Y2QYX6JMEMIPXAYU6ZGB662F6PZGGY6W456MNN4H2Q, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.305049777 (106h2m21.736129908s) [noise=true , address=104.36.82.35:4585 , piece_id=EA7XTERIGFLJMV33KWPJJEFXKDNYHSVJIBYES33AANWX6RFO3Z4A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.319397510 (106h2m21.750477631s) [noise=true , address=104.36.82.21:4488 , piece_id=ZJC77OAGWGPQHJLXGSY2JUS4POTFI4UYW2ORN2OET4U5QCCK2RYA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.324171042 (106h2m21.755251173s) [noise=true , address=104.36.82.8:4988 , piece_id=KZO4HIEGBYBTXIUMP7YETN5WBBNS2ZMGOA3W6H377RF5RV3RIGRQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.325044093 (106h2m21.756124224s) [noise=true , address=104.36.82.39:4758 , piece_id=6GQJIBOJ6XK6RILWWEPLNQH7OPKSYWFMQG3SXYTVONZRLV3YDU7A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.326189485 (106h2m21.757269616s) [noise=true , address=104.36.82.4:4833 , piece_id=WOJB675NZW5MONNZ7MHECJEYFRFUQDDQFCG4EATF2DTTQOY5FPIA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.358731589 (106h2m21.78981172s) [noise=true , address=104.36.82.15:4267 , piece_id=YOBGDLVCGO7Z6XLQQ6K7INK36KDGOHZNA6L2ROJF2VVC3YOE5CEQ, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.364728292 (106h2m21.795808413s) [noise=true , address=104.36.82.37:4686 , piece_id=TWWJUP6LY7VX4BW7PSTTZK53JTGSVBP2Y6NQCR3PXT44WCLGY2WA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.386279758 (106h2m21.817359889s) [noise=true , address=77.90.31.21:4906 , piece_id=FLIZ63F7NOFHDUEDYOAGMMGEGS473UQPXY2C3WKX67MGWJJEZBIA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.393790007 (106h2m21.824870158s) [noise=true , address=104.36.82.44:4876 , piece_id=576P5WMAQ6BCLTG4UAJX2ZCHGAMKCD47Z6YQBWJKHI5MQGGXT4XA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: <nil> Aug 13 03:28:27.393863398 (106h2m21.824943529s) [seg_pos={0 0}, upload=110366]: Segment reached optimal threshold of 14 pieces. Aug 13 03:28:27.393993957 (106h2m21.825074088s) [noise=true , address=104.36.82.46:4496 , piece_id=K6J6AEVYWM63WM6IFDXSGBUWK6WX72R26RLVLRZ64VSF4H2744JA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394051778 (106h2m21.82513191s) [noise=true , address=104.36.82.5:4866 , piece_id=22IXIQNEVDD6VILGPIQMS46JFYVQDI3BNI633KABQLCDBQQB22ZA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1zUcYfJE8xRW2Tow7bGB3PeNrVBUzVWR82JADkVwPdSr4CTPyr): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394083198 (106h2m21.82516333s) [noise=true , address=104.36.82.60:4363 , piece_id=SRLXSTGJIEBAM4H6UE6AF32SJY7N2ZHJPPDOXKTUN44ZI3LT2I5A, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7): protocol: expected piece hash; write tcp 188.166.225.170:42920->104.36.82.60:4363: use of closed network connection; write tcp 188.166.225.170:42920->104.36.82.60:4363: use of closed network connection; EOF Aug 13 03:28:27.394101453 (106h2m21.825181574s) [noise=true , address=79.127.220.98:4422 , piece_id=R6WFYRTY4IEQ4ORVBDWP5FBP676QRCJDPKSBNJDXL64C2OH2SWDA, seg_pos={0 0}, upload=110366]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MScXvmPpGZ6HRRwaoBMrZR3TvhDQh4nZkpSYggRQGzP1JbyVE): protocol: expected piece hash; context canceled; EOF Aug 13 03:28:27.394138204 (106h2m21.825218335s) [seg_pos={0 0}, upload=110366]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 13 03:28:27.394212385 (106h2m21.825292516s) [upload=110366]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 13 03:28:27.394217095 (106h2m21.825297226s) [upload=110366]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:28:27.720702752 (106h2m22.151782883s) [upload=110366]: Done uploading segments... Aug 13 03:28:27.720720255 (106h2m22.151800386s) [upload=110366]: Upload finished. err: <nil> Duration: 3.98759488s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 1.89419676s FAIL (2025-08-13 03:28:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 13 03:23:34.987469695 (105h57m29.418549826s) [upload=110284]: Starting upload Aug 13 03:23:34.987520002 (105h57m29.418600133s) [upload=110284]: Done starting upload Aug 13 03:23:34.987570067 (105h57m29.418650198s) [upload=110284]: Uploading segments... Aug 13 03:23:34.987604243 (105h57m29.418684374s) [upload=110284]: Got next segment. Inline: true Aug 13 03:23:34.987608641 (105h57m29.418688772s) [upload=110284]: Waiting for error group managing segments... Aug 13 03:23:34.987640973 (105h57m29.418721094s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:23:34.987644710 (105h57m29.418724841s) [upload=110284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:23:34.987647044 (105h57m29.418727165s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:23:35.199946062 (105h57m29.631026193s) [upload=110284]: Done uploading segments... Aug 13 03:23:35.199964076 (105h57m29.631044207s) [upload=110284]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.894216597s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 1.928809465s FAIL (2025-08-13 03:27:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 13 03:06:41.898033585 (105h40m36.329113716s) [upload=110007]: Starting upload Aug 13 03:06:41.898093971 (105h40m36.329174102s) [upload=110007]: Done starting upload Aug 13 03:06:41.898140650 (105h40m36.329220781s) [upload=110007]: Uploading segments... Aug 13 03:06:41.898172331 (105h40m36.329252462s) [upload=110007]: Got next segment. Inline: true Aug 13 03:06:41.898176078 (105h40m36.329256209s) [upload=110007]: Waiting for error group managing segments... Aug 13 03:06:41.898208831 (105h40m36.329288962s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:06:41.898212778 (105h40m36.329292909s) [upload=110007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:06:41.898215563 (105h40m36.329295694s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:07:11.244546166 (105h41m5.675626307s) [upload=110007]: Done uploading segments... Aug 13 03:07:11.244585330 (105h41m5.675665482s) [upload=110007]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.928834994s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 1.908976159s FAIL (2025-08-13 03:27:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 13 03:23:34.987469695 (105h57m29.418549826s) [upload=110284]: Starting upload Aug 13 03:23:34.987520002 (105h57m29.418600133s) [upload=110284]: Done starting upload Aug 13 03:23:34.987570067 (105h57m29.418650198s) [upload=110284]: Uploading segments... Aug 13 03:23:34.987604243 (105h57m29.418684374s) [upload=110284]: Got next segment. Inline: true Aug 13 03:23:34.987608641 (105h57m29.418688772s) [upload=110284]: Waiting for error group managing segments... Aug 13 03:23:34.987640973 (105h57m29.418721094s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:23:34.987644710 (105h57m29.418724841s) [upload=110284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:23:34.987647044 (105h57m29.418727165s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:23:35.199946062 (105h57m29.631026193s) [upload=110284]: Done uploading segments... Aug 13 03:23:35.199964076 (105h57m29.631044207s) [upload=110284]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.909009523s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................................X.....XX..........] duration: 7.3616121s FAIL (2025-08-13 03:27:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.36163322s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X..X.......X...........X...............X...................] duration: 5.815592076s FAIL (2025-08-13 03:27:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 07a64daf5ce9db32d41aea6196fcc70ddc5670c6fb7b746f966c0a14f0f90cad != before e0a47ea874bf973a2cf983e7ff42ad385b8577ce8b4f3c6f68efa1aa67839818 ======LOGS====== Aug 13 03:14:27.200063120 (105h48m21.631143251s) [upload=110132]: Starting upload Aug 13 03:14:27.200107645 (105h48m21.631187776s) [upload=110132]: Done starting upload Aug 13 03:14:27.200237103 (105h48m21.631317234s) [upload=110132]: Uploading segments... Aug 13 03:14:27.200364997 (105h48m21.631445128s) [upload=110132]: Got next segment. Inline: false Aug 13 03:14:27.200415604 (105h48m21.631495785s) [upload=110132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:14:27.200424100 (105h48m21.631504271s) [upload=110132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 13 03:14:27.660811382 (105h48m22.091891513s) [seg_pos={0 0}, upload=110132]: Begin upload segment... Aug 13 03:14:27.660865877 (105h48m22.091946008s) [seg_pos={0 0}, upload=110132]: Done begin upload segment. Aug 13 03:14:27.660886587 (105h48m22.091966718s) [noise=true , address=77.90.31.10:4528 , piece_id=XKSFJ2I7HYD72TOTWFHDRRLTWE6SFSMKTDVKYIYFWI6VYQF4EW5Q, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.660900844 (105h48m22.091981055s) [noise=true , address=79.127.219.38:4035 , piece_id=MXSRDYRUPY6DSE6N66WNQZPCC6KVBHISPOIZBGFAQSD3CUGSFWOA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.661384269 (105h48m22.0924644s) [noise=true , address=104.36.82.58:4284 , piece_id=K5J626LGT3TOFW6Q4G22RJJYQGXPYS2B74RB34HVPD6HTOQA2QCA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.661598819 (105h48m22.09267894s) [noise=true , address=104.36.82.14:4250 , piece_id=CQY4HBMBDYVUKR4637GSB33VW6G4OEP53VH27DMSBPS54CATSBWA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.661669895 (105h48m22.092750026s) [noise=true , address=104.36.82.47:4544 , piece_id=TIKP5LMX65MWSBUUPMX36JYRVFC3JVCA5KLRS7CIZMF3SJVS73OA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.661996209 (105h48m22.09307634s) [noise=true , address=77.90.31.14:4679 , piece_id=CFPI32OHVSZTT2YEOSL5ZXAHA37PT57VMTTTCTTIRQVMXAJM6D5A, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.662169181 (105h48m22.093249342s) [noise=true , address=77.90.31.10:4530 , piece_id=JILOKHNOQCGIDXTL2NQF4HCNYE6RJXNH4OICDIVSHADVBIUVA4ZA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.662275895 (105h48m22.093356016s) [noise=true , address=104.36.82.18:4377 , piece_id=Y7NIH46TSTZBIMNAK6WLRFX6WYAZR2BPQLVKHJ4OK44TTD3JP2AA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.662522045 (105h48m22.093602166s) [noise=true , address=104.36.82.9:4024 , piece_id=Y5CY7VLCSC6BNGIH6NF6446BOU4LNIX5GCCY6DSWCPKXPCK4QDQQ, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.662779709 (105h48m22.09385984s) [noise=true , address=104.36.82.46:4484 , piece_id=YRE57GHVKPCOHVRDGSL4FGGX4NPXUJF4L7ELHZGHJ4EC4QDMEU3A, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.662985592 (105h48m22.094065723s) [noise=true , address=104.36.82.59:4312 , piece_id=FZB6SMU32MTZTBAHUTHLWVZGQ53C67FAAIIH6VE2MTLS6PGFXXZQ, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.663176938 (105h48m22.094257069s) [noise=true , address=104.36.82.60:4352 , piece_id=UCDAKY7PVPM2LXMCEZYWZNQJV6DJRZIBB2VMJP2IB6N4ZEU6Q7IA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.663367453 (105h48m22.094447584s) [noise=true , address=104.36.82.16:4299 , piece_id=CTNAIZNCPYLZLJZSH7OVSKBIJ2Q3J2QC75ONHCNPKVGRQEQWLL2Q, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.663568779 (105h48m22.09464891s) [noise=true , address=104.36.82.37:4678 , piece_id=6JTI7SDS4DTDJFAF4FZCCCOON7KOPOXXT5MZ2MDIPSQXZZKF6TKQ, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.663767839 (105h48m22.09484797s) [noise=true , address=104.36.82.36:4637 , piece_id=QXP45IJF7253J7A5MY3C4DAA4VFFT5XULJ3Y4QTXTDCTVJP4IYGA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.663960167 (105h48m22.095040288s) [noise=true , address=104.36.82.23:4472 , piece_id=IRJTNTCAIGVWU4HIE3L3ZCZH6VM6FHFU3SEZDXFYEV4R2BAP45RA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.664144580 (105h48m22.095224711s) [noise=true , address=104.36.82.17:4329 , piece_id=KPGYZK5QIGRNMUHQLFRGRNIPI5EONZBHCGARRSDZCK2UFJLNN4TA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.664326608 (105h48m22.095406739s) [noise=true , address=104.36.82.11:4138 , piece_id=7L7M5DMAFNHOHVMUAQPFOKLU4TKIBL3TMS6D2F6X5T23BKDKGS7Q, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:29.202132828 (105h48m23.633212949s) [upload=110132]: Next returned nil segment Aug 13 03:14:29.202151514 (105h48m23.633231645s) [upload=110132]: Waiting for error group managing segments... Aug 13 03:14:29.624905241 (105h48m24.055985372s) [noise=true , address=77.90.31.14:4679 , piece_id=CFPI32OHVSZTT2YEOSL5ZXAHA37PT57VMTTTCTTIRQVMXAJM6D5A, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.625120442 (105h48m24.056200574s) [noise=true , address=104.36.82.59:4312 , piece_id=FZB6SMU32MTZTBAHUTHLWVZGQ53C67FAAIIH6VE2MTLS6PGFXXZQ, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.667080856 (105h48m24.098160987s) [noise=true , address=104.36.82.9:4024 , piece_id=Y5CY7VLCSC6BNGIH6NF6446BOU4LNIX5GCCY6DSWCPKXPCK4QDQQ, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.669032951 (105h48m24.100113082s) [noise=true , address=104.36.82.14:4250 , piece_id=CQY4HBMBDYVUKR4637GSB33VW6G4OEP53VH27DMSBPS54CATSBWA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.699132833 (105h48m24.130212964s) [noise=true , address=104.36.82.11:4138 , piece_id=7L7M5DMAFNHOHVMUAQPFOKLU4TKIBL3TMS6D2F6X5T23BKDKGS7Q, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.717982878 (105h48m24.149063009s) [noise=true , address=104.36.82.36:4637 , piece_id=QXP45IJF7253J7A5MY3C4DAA4VFFT5XULJ3Y4QTXTDCTVJP4IYGA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.724446195 (105h48m24.155526316s) [noise=true , address=104.36.82.37:4678 , piece_id=6JTI7SDS4DTDJFAF4FZCCCOON7KOPOXXT5MZ2MDIPSQXZZKF6TKQ, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.731783564 (105h48m24.162863695s) [noise=true , address=104.36.82.58:4284 , piece_id=K5J626LGT3TOFW6Q4G22RJJYQGXPYS2B74RB34HVPD6HTOQA2QCA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.780951378 (105h48m24.212031509s) [noise=true , address=104.36.82.46:4484 , piece_id=YRE57GHVKPCOHVRDGSL4FGGX4NPXUJF4L7ELHZGHJ4EC4QDMEU3A, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.794319577 (105h48m24.225399708s) [noise=true , address=104.36.82.47:4544 , piece_id=TIKP5LMX65MWSBUUPMX36JYRVFC3JVCA5KLRS7CIZMF3SJVS73OA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:30.252036350 (105h48m24.683116481s) [noise=true , address=79.127.219.38:4035 , piece_id=MXSRDYRUPY6DSE6N66WNQZPCC6KVBHISPOIZBGFAQSD3CUGSFWOA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:30.523524634 (105h48m24.954604765s) [noise=true , address=104.36.82.23:4472 , piece_id=IRJTNTCAIGVWU4HIE3L3ZCZH6VM6FHFU3SEZDXFYEV4R2BAP45RA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:31.207986103 (105h48m25.639066235s) [noise=true , address=77.90.31.10:4528 , piece_id=XKSFJ2I7HYD72TOTWFHDRRLTWE6SFSMKTDVKYIYFWI6VYQF4EW5Q, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:31.757453767 (105h48m26.188533898s) [noise=true , address=104.36.82.16:4299 , piece_id=CTNAIZNCPYLZLJZSH7OVSKBIJ2Q3J2QC75ONHCNPKVGRQEQWLL2Q, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:31.757477813 (105h48m26.188557934s) [seg_pos={0 0}, upload=110132]: Segment reached optimal threshold of 14 pieces. Aug 13 03:14:31.757561022 (105h48m26.188641153s) [noise=true , address=77.90.31.10:4530 , piece_id=JILOKHNOQCGIDXTL2NQF4HCNYE6RJXNH4OICDIVSHADVBIUVA4ZA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne): protocol: expected piece hash; context canceled; EOF Aug 13 03:14:31.757627769 (105h48m26.1887079s) [noise=true , address=104.36.82.18:4377 , piece_id=Y7NIH46TSTZBIMNAK6WLRFX6WYAZR2BPQLVKHJ4OK44TTD3JP2AA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JHy4o8dxFNeeiitLPavxspcf4rL9KfckcRAuC121v8YZG6qmT): protocol: expected piece hash; context canceled; EOF Aug 13 03:14:31.757682684 (105h48m26.188762815s) [noise=true , address=104.36.82.17:4329 , piece_id=KPGYZK5QIGRNMUHQLFRGRNIPI5EONZBHCGARRSDZCK2UFJLNN4TA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: expected piece hash; context canceled; EOF Aug 13 03:14:31.757703594 (105h48m26.188783725s) [noise=true , address=104.36.82.60:4352 , piece_id=UCDAKY7PVPM2LXMCEZYWZNQJV6DJRZIBB2VMJP2IB6N4ZEU6Q7IA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: write tcp 188.166.225.170:47910->104.36.82.60:4352: use of closed network connection; write tcp 188.166.225.170:47910->104.36.82.60:4352: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47910->104.36.82.60:4352: use of closed network connection Aug 13 03:14:31.757749713 (105h48m26.188829844s) [seg_pos={0 0}, upload=110132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 13 03:14:31.757833714 (105h48m26.188913845s) [upload=110132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 13 03:14:31.757837731 (105h48m26.188917862s) [upload=110132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:14:32.059045060 (105h48m26.490125191s) [upload=110132]: Done uploading segments... Aug 13 03:14:32.059061040 (105h48m26.490141171s) [upload=110132]: Upload finished. err: <nil> Duration: 5.815626332s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.675256167s FAIL (2025-08-13 03:27:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 13 03:06:41.898033585 (105h40m36.329113716s) [upload=110007]: Starting upload Aug 13 03:06:41.898093971 (105h40m36.329174102s) [upload=110007]: Done starting upload Aug 13 03:06:41.898140650 (105h40m36.329220781s) [upload=110007]: Uploading segments... Aug 13 03:06:41.898172331 (105h40m36.329252462s) [upload=110007]: Got next segment. Inline: true Aug 13 03:06:41.898176078 (105h40m36.329256209s) [upload=110007]: Waiting for error group managing segments... Aug 13 03:06:41.898208831 (105h40m36.329288962s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:06:41.898212778 (105h40m36.329292909s) [upload=110007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:06:41.898215563 (105h40m36.329295694s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:07:11.244546166 (105h41m5.675626307s) [upload=110007]: Done uploading segments... Aug 13 03:07:11.244585330 (105h41m5.675665482s) [upload=110007]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.675285234s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 1.952342572s FAIL (2025-08-13 03:26:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 13 03:23:34.987469695 (105h57m29.418549826s) [upload=110284]: Starting upload Aug 13 03:23:34.987520002 (105h57m29.418600133s) [upload=110284]: Done starting upload Aug 13 03:23:34.987570067 (105h57m29.418650198s) [upload=110284]: Uploading segments... Aug 13 03:23:34.987604243 (105h57m29.418684374s) [upload=110284]: Got next segment. Inline: true Aug 13 03:23:34.987608641 (105h57m29.418688772s) [upload=110284]: Waiting for error group managing segments... Aug 13 03:23:34.987640973 (105h57m29.418721094s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:23:34.987644710 (105h57m29.418724841s) [upload=110284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:23:34.987647044 (105h57m29.418727165s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:23:35.199946062 (105h57m29.631026193s) [upload=110284]: Done uploading segments... Aug 13 03:23:35.199964076 (105h57m29.631044207s) [upload=110284]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.952369555s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................................X.....XX.........] duration: 7.666927174s FAIL (2025-08-13 03:26:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.666943526s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X..X.......X...........X...............X..................] duration: 5.594668226s FAIL (2025-08-13 03:26:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 07a64daf5ce9db32d41aea6196fcc70ddc5670c6fb7b746f966c0a14f0f90cad != before e0a47ea874bf973a2cf983e7ff42ad385b8577ce8b4f3c6f68efa1aa67839818 ======LOGS====== Aug 13 03:14:27.200063120 (105h48m21.631143251s) [upload=110132]: Starting upload Aug 13 03:14:27.200107645 (105h48m21.631187776s) [upload=110132]: Done starting upload Aug 13 03:14:27.200237103 (105h48m21.631317234s) [upload=110132]: Uploading segments... Aug 13 03:14:27.200364997 (105h48m21.631445128s) [upload=110132]: Got next segment. Inline: false Aug 13 03:14:27.200415604 (105h48m21.631495785s) [upload=110132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:14:27.200424100 (105h48m21.631504271s) [upload=110132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 13 03:14:27.660811382 (105h48m22.091891513s) [seg_pos={0 0}, upload=110132]: Begin upload segment... Aug 13 03:14:27.660865877 (105h48m22.091946008s) [seg_pos={0 0}, upload=110132]: Done begin upload segment. Aug 13 03:14:27.660886587 (105h48m22.091966718s) [noise=true , address=77.90.31.10:4528 , piece_id=XKSFJ2I7HYD72TOTWFHDRRLTWE6SFSMKTDVKYIYFWI6VYQF4EW5Q, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.660900844 (105h48m22.091981055s) [noise=true , address=79.127.219.38:4035 , piece_id=MXSRDYRUPY6DSE6N66WNQZPCC6KVBHISPOIZBGFAQSD3CUGSFWOA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.661384269 (105h48m22.0924644s) [noise=true , address=104.36.82.58:4284 , piece_id=K5J626LGT3TOFW6Q4G22RJJYQGXPYS2B74RB34HVPD6HTOQA2QCA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.661598819 (105h48m22.09267894s) [noise=true , address=104.36.82.14:4250 , piece_id=CQY4HBMBDYVUKR4637GSB33VW6G4OEP53VH27DMSBPS54CATSBWA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.661669895 (105h48m22.092750026s) [noise=true , address=104.36.82.47:4544 , piece_id=TIKP5LMX65MWSBUUPMX36JYRVFC3JVCA5KLRS7CIZMF3SJVS73OA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.661996209 (105h48m22.09307634s) [noise=true , address=77.90.31.14:4679 , piece_id=CFPI32OHVSZTT2YEOSL5ZXAHA37PT57VMTTTCTTIRQVMXAJM6D5A, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.662169181 (105h48m22.093249342s) [noise=true , address=77.90.31.10:4530 , piece_id=JILOKHNOQCGIDXTL2NQF4HCNYE6RJXNH4OICDIVSHADVBIUVA4ZA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.662275895 (105h48m22.093356016s) [noise=true , address=104.36.82.18:4377 , piece_id=Y7NIH46TSTZBIMNAK6WLRFX6WYAZR2BPQLVKHJ4OK44TTD3JP2AA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.662522045 (105h48m22.093602166s) [noise=true , address=104.36.82.9:4024 , piece_id=Y5CY7VLCSC6BNGIH6NF6446BOU4LNIX5GCCY6DSWCPKXPCK4QDQQ, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.662779709 (105h48m22.09385984s) [noise=true , address=104.36.82.46:4484 , piece_id=YRE57GHVKPCOHVRDGSL4FGGX4NPXUJF4L7ELHZGHJ4EC4QDMEU3A, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.662985592 (105h48m22.094065723s) [noise=true , address=104.36.82.59:4312 , piece_id=FZB6SMU32MTZTBAHUTHLWVZGQ53C67FAAIIH6VE2MTLS6PGFXXZQ, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.663176938 (105h48m22.094257069s) [noise=true , address=104.36.82.60:4352 , piece_id=UCDAKY7PVPM2LXMCEZYWZNQJV6DJRZIBB2VMJP2IB6N4ZEU6Q7IA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.663367453 (105h48m22.094447584s) [noise=true , address=104.36.82.16:4299 , piece_id=CTNAIZNCPYLZLJZSH7OVSKBIJ2Q3J2QC75ONHCNPKVGRQEQWLL2Q, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.663568779 (105h48m22.09464891s) [noise=true , address=104.36.82.37:4678 , piece_id=6JTI7SDS4DTDJFAF4FZCCCOON7KOPOXXT5MZ2MDIPSQXZZKF6TKQ, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.663767839 (105h48m22.09484797s) [noise=true , address=104.36.82.36:4637 , piece_id=QXP45IJF7253J7A5MY3C4DAA4VFFT5XULJ3Y4QTXTDCTVJP4IYGA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.663960167 (105h48m22.095040288s) [noise=true , address=104.36.82.23:4472 , piece_id=IRJTNTCAIGVWU4HIE3L3ZCZH6VM6FHFU3SEZDXFYEV4R2BAP45RA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.664144580 (105h48m22.095224711s) [noise=true , address=104.36.82.17:4329 , piece_id=KPGYZK5QIGRNMUHQLFRGRNIPI5EONZBHCGARRSDZCK2UFJLNN4TA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.664326608 (105h48m22.095406739s) [noise=true , address=104.36.82.11:4138 , piece_id=7L7M5DMAFNHOHVMUAQPFOKLU4TKIBL3TMS6D2F6X5T23BKDKGS7Q, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:29.202132828 (105h48m23.633212949s) [upload=110132]: Next returned nil segment Aug 13 03:14:29.202151514 (105h48m23.633231645s) [upload=110132]: Waiting for error group managing segments... Aug 13 03:14:29.624905241 (105h48m24.055985372s) [noise=true , address=77.90.31.14:4679 , piece_id=CFPI32OHVSZTT2YEOSL5ZXAHA37PT57VMTTTCTTIRQVMXAJM6D5A, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.625120442 (105h48m24.056200574s) [noise=true , address=104.36.82.59:4312 , piece_id=FZB6SMU32MTZTBAHUTHLWVZGQ53C67FAAIIH6VE2MTLS6PGFXXZQ, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.667080856 (105h48m24.098160987s) [noise=true , address=104.36.82.9:4024 , piece_id=Y5CY7VLCSC6BNGIH6NF6446BOU4LNIX5GCCY6DSWCPKXPCK4QDQQ, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.669032951 (105h48m24.100113082s) [noise=true , address=104.36.82.14:4250 , piece_id=CQY4HBMBDYVUKR4637GSB33VW6G4OEP53VH27DMSBPS54CATSBWA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.699132833 (105h48m24.130212964s) [noise=true , address=104.36.82.11:4138 , piece_id=7L7M5DMAFNHOHVMUAQPFOKLU4TKIBL3TMS6D2F6X5T23BKDKGS7Q, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.717982878 (105h48m24.149063009s) [noise=true , address=104.36.82.36:4637 , piece_id=QXP45IJF7253J7A5MY3C4DAA4VFFT5XULJ3Y4QTXTDCTVJP4IYGA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.724446195 (105h48m24.155526316s) [noise=true , address=104.36.82.37:4678 , piece_id=6JTI7SDS4DTDJFAF4FZCCCOON7KOPOXXT5MZ2MDIPSQXZZKF6TKQ, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.731783564 (105h48m24.162863695s) [noise=true , address=104.36.82.58:4284 , piece_id=K5J626LGT3TOFW6Q4G22RJJYQGXPYS2B74RB34HVPD6HTOQA2QCA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.780951378 (105h48m24.212031509s) [noise=true , address=104.36.82.46:4484 , piece_id=YRE57GHVKPCOHVRDGSL4FGGX4NPXUJF4L7ELHZGHJ4EC4QDMEU3A, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.794319577 (105h48m24.225399708s) [noise=true , address=104.36.82.47:4544 , piece_id=TIKP5LMX65MWSBUUPMX36JYRVFC3JVCA5KLRS7CIZMF3SJVS73OA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:30.252036350 (105h48m24.683116481s) [noise=true , address=79.127.219.38:4035 , piece_id=MXSRDYRUPY6DSE6N66WNQZPCC6KVBHISPOIZBGFAQSD3CUGSFWOA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:30.523524634 (105h48m24.954604765s) [noise=true , address=104.36.82.23:4472 , piece_id=IRJTNTCAIGVWU4HIE3L3ZCZH6VM6FHFU3SEZDXFYEV4R2BAP45RA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:31.207986103 (105h48m25.639066235s) [noise=true , address=77.90.31.10:4528 , piece_id=XKSFJ2I7HYD72TOTWFHDRRLTWE6SFSMKTDVKYIYFWI6VYQF4EW5Q, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:31.757453767 (105h48m26.188533898s) [noise=true , address=104.36.82.16:4299 , piece_id=CTNAIZNCPYLZLJZSH7OVSKBIJ2Q3J2QC75ONHCNPKVGRQEQWLL2Q, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:31.757477813 (105h48m26.188557934s) [seg_pos={0 0}, upload=110132]: Segment reached optimal threshold of 14 pieces. Aug 13 03:14:31.757561022 (105h48m26.188641153s) [noise=true , address=77.90.31.10:4530 , piece_id=JILOKHNOQCGIDXTL2NQF4HCNYE6RJXNH4OICDIVSHADVBIUVA4ZA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne): protocol: expected piece hash; context canceled; EOF Aug 13 03:14:31.757627769 (105h48m26.1887079s) [noise=true , address=104.36.82.18:4377 , piece_id=Y7NIH46TSTZBIMNAK6WLRFX6WYAZR2BPQLVKHJ4OK44TTD3JP2AA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JHy4o8dxFNeeiitLPavxspcf4rL9KfckcRAuC121v8YZG6qmT): protocol: expected piece hash; context canceled; EOF Aug 13 03:14:31.757682684 (105h48m26.188762815s) [noise=true , address=104.36.82.17:4329 , piece_id=KPGYZK5QIGRNMUHQLFRGRNIPI5EONZBHCGARRSDZCK2UFJLNN4TA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: expected piece hash; context canceled; EOF Aug 13 03:14:31.757703594 (105h48m26.188783725s) [noise=true , address=104.36.82.60:4352 , piece_id=UCDAKY7PVPM2LXMCEZYWZNQJV6DJRZIBB2VMJP2IB6N4ZEU6Q7IA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: write tcp 188.166.225.170:47910->104.36.82.60:4352: use of closed network connection; write tcp 188.166.225.170:47910->104.36.82.60:4352: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47910->104.36.82.60:4352: use of closed network connection Aug 13 03:14:31.757749713 (105h48m26.188829844s) [seg_pos={0 0}, upload=110132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 13 03:14:31.757833714 (105h48m26.188913845s) [upload=110132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 13 03:14:31.757837731 (105h48m26.188917862s) [upload=110132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:14:32.059045060 (105h48m26.490125191s) [upload=110132]: Done uploading segments... Aug 13 03:14:32.059061040 (105h48m26.490141171s) [upload=110132]: Upload finished. err: <nil> Duration: 5.594699016s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................................X.....XX........] duration: 7.597547819s FAIL (2025-08-13 03:26:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.597578558s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 1.883864229s FAIL (2025-08-13 03:26:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 13 03:06:41.898033585 (105h40m36.329113716s) [upload=110007]: Starting upload Aug 13 03:06:41.898093971 (105h40m36.329174102s) [upload=110007]: Done starting upload Aug 13 03:06:41.898140650 (105h40m36.329220781s) [upload=110007]: Uploading segments... Aug 13 03:06:41.898172331 (105h40m36.329252462s) [upload=110007]: Got next segment. Inline: true Aug 13 03:06:41.898176078 (105h40m36.329256209s) [upload=110007]: Waiting for error group managing segments... Aug 13 03:06:41.898208831 (105h40m36.329288962s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:06:41.898212778 (105h40m36.329292909s) [upload=110007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:06:41.898215563 (105h40m36.329295694s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:07:11.244546166 (105h41m5.675626307s) [upload=110007]: Done uploading segments... Aug 13 03:07:11.244585330 (105h41m5.675665482s) [upload=110007]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.883895538s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 1.844786691s FAIL (2025-08-13 03:26:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 13 03:23:34.987469695 (105h57m29.418549826s) [upload=110284]: Starting upload Aug 13 03:23:34.987520002 (105h57m29.418600133s) [upload=110284]: Done starting upload Aug 13 03:23:34.987570067 (105h57m29.418650198s) [upload=110284]: Uploading segments... Aug 13 03:23:34.987604243 (105h57m29.418684374s) [upload=110284]: Got next segment. Inline: true Aug 13 03:23:34.987608641 (105h57m29.418688772s) [upload=110284]: Waiting for error group managing segments... Aug 13 03:23:34.987640973 (105h57m29.418721094s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:23:34.987644710 (105h57m29.418724841s) [upload=110284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:23:34.987647044 (105h57m29.418727165s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:23:35.199946062 (105h57m29.631026193s) [upload=110284]: Done uploading segments... Aug 13 03:23:35.199964076 (105h57m29.631044207s) [upload=110284]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.844810878s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X..X.......X...........X...............X.................] duration: 6.133650999s FAIL (2025-08-13 03:25:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 07a64daf5ce9db32d41aea6196fcc70ddc5670c6fb7b746f966c0a14f0f90cad != before e0a47ea874bf973a2cf983e7ff42ad385b8577ce8b4f3c6f68efa1aa67839818 ======LOGS====== Aug 13 03:14:27.200063120 (105h48m21.631143251s) [upload=110132]: Starting upload Aug 13 03:14:27.200107645 (105h48m21.631187776s) [upload=110132]: Done starting upload Aug 13 03:14:27.200237103 (105h48m21.631317234s) [upload=110132]: Uploading segments... Aug 13 03:14:27.200364997 (105h48m21.631445128s) [upload=110132]: Got next segment. Inline: false Aug 13 03:14:27.200415604 (105h48m21.631495785s) [upload=110132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:14:27.200424100 (105h48m21.631504271s) [upload=110132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 13 03:14:27.660811382 (105h48m22.091891513s) [seg_pos={0 0}, upload=110132]: Begin upload segment... Aug 13 03:14:27.660865877 (105h48m22.091946008s) [seg_pos={0 0}, upload=110132]: Done begin upload segment. Aug 13 03:14:27.660886587 (105h48m22.091966718s) [noise=true , address=77.90.31.10:4528 , piece_id=XKSFJ2I7HYD72TOTWFHDRRLTWE6SFSMKTDVKYIYFWI6VYQF4EW5Q, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.660900844 (105h48m22.091981055s) [noise=true , address=79.127.219.38:4035 , piece_id=MXSRDYRUPY6DSE6N66WNQZPCC6KVBHISPOIZBGFAQSD3CUGSFWOA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.661384269 (105h48m22.0924644s) [noise=true , address=104.36.82.58:4284 , piece_id=K5J626LGT3TOFW6Q4G22RJJYQGXPYS2B74RB34HVPD6HTOQA2QCA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.661598819 (105h48m22.09267894s) [noise=true , address=104.36.82.14:4250 , piece_id=CQY4HBMBDYVUKR4637GSB33VW6G4OEP53VH27DMSBPS54CATSBWA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.661669895 (105h48m22.092750026s) [noise=true , address=104.36.82.47:4544 , piece_id=TIKP5LMX65MWSBUUPMX36JYRVFC3JVCA5KLRS7CIZMF3SJVS73OA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.661996209 (105h48m22.09307634s) [noise=true , address=77.90.31.14:4679 , piece_id=CFPI32OHVSZTT2YEOSL5ZXAHA37PT57VMTTTCTTIRQVMXAJM6D5A, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.662169181 (105h48m22.093249342s) [noise=true , address=77.90.31.10:4530 , piece_id=JILOKHNOQCGIDXTL2NQF4HCNYE6RJXNH4OICDIVSHADVBIUVA4ZA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.662275895 (105h48m22.093356016s) [noise=true , address=104.36.82.18:4377 , piece_id=Y7NIH46TSTZBIMNAK6WLRFX6WYAZR2BPQLVKHJ4OK44TTD3JP2AA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.662522045 (105h48m22.093602166s) [noise=true , address=104.36.82.9:4024 , piece_id=Y5CY7VLCSC6BNGIH6NF6446BOU4LNIX5GCCY6DSWCPKXPCK4QDQQ, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.662779709 (105h48m22.09385984s) [noise=true , address=104.36.82.46:4484 , piece_id=YRE57GHVKPCOHVRDGSL4FGGX4NPXUJF4L7ELHZGHJ4EC4QDMEU3A, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.662985592 (105h48m22.094065723s) [noise=true , address=104.36.82.59:4312 , piece_id=FZB6SMU32MTZTBAHUTHLWVZGQ53C67FAAIIH6VE2MTLS6PGFXXZQ, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.663176938 (105h48m22.094257069s) [noise=true , address=104.36.82.60:4352 , piece_id=UCDAKY7PVPM2LXMCEZYWZNQJV6DJRZIBB2VMJP2IB6N4ZEU6Q7IA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.663367453 (105h48m22.094447584s) [noise=true , address=104.36.82.16:4299 , piece_id=CTNAIZNCPYLZLJZSH7OVSKBIJ2Q3J2QC75ONHCNPKVGRQEQWLL2Q, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.663568779 (105h48m22.09464891s) [noise=true , address=104.36.82.37:4678 , piece_id=6JTI7SDS4DTDJFAF4FZCCCOON7KOPOXXT5MZ2MDIPSQXZZKF6TKQ, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.663767839 (105h48m22.09484797s) [noise=true , address=104.36.82.36:4637 , piece_id=QXP45IJF7253J7A5MY3C4DAA4VFFT5XULJ3Y4QTXTDCTVJP4IYGA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.663960167 (105h48m22.095040288s) [noise=true , address=104.36.82.23:4472 , piece_id=IRJTNTCAIGVWU4HIE3L3ZCZH6VM6FHFU3SEZDXFYEV4R2BAP45RA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.664144580 (105h48m22.095224711s) [noise=true , address=104.36.82.17:4329 , piece_id=KPGYZK5QIGRNMUHQLFRGRNIPI5EONZBHCGARRSDZCK2UFJLNN4TA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.664326608 (105h48m22.095406739s) [noise=true , address=104.36.82.11:4138 , piece_id=7L7M5DMAFNHOHVMUAQPFOKLU4TKIBL3TMS6D2F6X5T23BKDKGS7Q, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:29.202132828 (105h48m23.633212949s) [upload=110132]: Next returned nil segment Aug 13 03:14:29.202151514 (105h48m23.633231645s) [upload=110132]: Waiting for error group managing segments... Aug 13 03:14:29.624905241 (105h48m24.055985372s) [noise=true , address=77.90.31.14:4679 , piece_id=CFPI32OHVSZTT2YEOSL5ZXAHA37PT57VMTTTCTTIRQVMXAJM6D5A, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.625120442 (105h48m24.056200574s) [noise=true , address=104.36.82.59:4312 , piece_id=FZB6SMU32MTZTBAHUTHLWVZGQ53C67FAAIIH6VE2MTLS6PGFXXZQ, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.667080856 (105h48m24.098160987s) [noise=true , address=104.36.82.9:4024 , piece_id=Y5CY7VLCSC6BNGIH6NF6446BOU4LNIX5GCCY6DSWCPKXPCK4QDQQ, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.669032951 (105h48m24.100113082s) [noise=true , address=104.36.82.14:4250 , piece_id=CQY4HBMBDYVUKR4637GSB33VW6G4OEP53VH27DMSBPS54CATSBWA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.699132833 (105h48m24.130212964s) [noise=true , address=104.36.82.11:4138 , piece_id=7L7M5DMAFNHOHVMUAQPFOKLU4TKIBL3TMS6D2F6X5T23BKDKGS7Q, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.717982878 (105h48m24.149063009s) [noise=true , address=104.36.82.36:4637 , piece_id=QXP45IJF7253J7A5MY3C4DAA4VFFT5XULJ3Y4QTXTDCTVJP4IYGA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.724446195 (105h48m24.155526316s) [noise=true , address=104.36.82.37:4678 , piece_id=6JTI7SDS4DTDJFAF4FZCCCOON7KOPOXXT5MZ2MDIPSQXZZKF6TKQ, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.731783564 (105h48m24.162863695s) [noise=true , address=104.36.82.58:4284 , piece_id=K5J626LGT3TOFW6Q4G22RJJYQGXPYS2B74RB34HVPD6HTOQA2QCA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.780951378 (105h48m24.212031509s) [noise=true , address=104.36.82.46:4484 , piece_id=YRE57GHVKPCOHVRDGSL4FGGX4NPXUJF4L7ELHZGHJ4EC4QDMEU3A, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.794319577 (105h48m24.225399708s) [noise=true , address=104.36.82.47:4544 , piece_id=TIKP5LMX65MWSBUUPMX36JYRVFC3JVCA5KLRS7CIZMF3SJVS73OA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:30.252036350 (105h48m24.683116481s) [noise=true , address=79.127.219.38:4035 , piece_id=MXSRDYRUPY6DSE6N66WNQZPCC6KVBHISPOIZBGFAQSD3CUGSFWOA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:30.523524634 (105h48m24.954604765s) [noise=true , address=104.36.82.23:4472 , piece_id=IRJTNTCAIGVWU4HIE3L3ZCZH6VM6FHFU3SEZDXFYEV4R2BAP45RA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:31.207986103 (105h48m25.639066235s) [noise=true , address=77.90.31.10:4528 , piece_id=XKSFJ2I7HYD72TOTWFHDRRLTWE6SFSMKTDVKYIYFWI6VYQF4EW5Q, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:31.757453767 (105h48m26.188533898s) [noise=true , address=104.36.82.16:4299 , piece_id=CTNAIZNCPYLZLJZSH7OVSKBIJ2Q3J2QC75ONHCNPKVGRQEQWLL2Q, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:31.757477813 (105h48m26.188557934s) [seg_pos={0 0}, upload=110132]: Segment reached optimal threshold of 14 pieces. Aug 13 03:14:31.757561022 (105h48m26.188641153s) [noise=true , address=77.90.31.10:4530 , piece_id=JILOKHNOQCGIDXTL2NQF4HCNYE6RJXNH4OICDIVSHADVBIUVA4ZA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne): protocol: expected piece hash; context canceled; EOF Aug 13 03:14:31.757627769 (105h48m26.1887079s) [noise=true , address=104.36.82.18:4377 , piece_id=Y7NIH46TSTZBIMNAK6WLRFX6WYAZR2BPQLVKHJ4OK44TTD3JP2AA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JHy4o8dxFNeeiitLPavxspcf4rL9KfckcRAuC121v8YZG6qmT): protocol: expected piece hash; context canceled; EOF Aug 13 03:14:31.757682684 (105h48m26.188762815s) [noise=true , address=104.36.82.17:4329 , piece_id=KPGYZK5QIGRNMUHQLFRGRNIPI5EONZBHCGARRSDZCK2UFJLNN4TA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: expected piece hash; context canceled; EOF Aug 13 03:14:31.757703594 (105h48m26.188783725s) [noise=true , address=104.36.82.60:4352 , piece_id=UCDAKY7PVPM2LXMCEZYWZNQJV6DJRZIBB2VMJP2IB6N4ZEU6Q7IA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: write tcp 188.166.225.170:47910->104.36.82.60:4352: use of closed network connection; write tcp 188.166.225.170:47910->104.36.82.60:4352: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47910->104.36.82.60:4352: use of closed network connection Aug 13 03:14:31.757749713 (105h48m26.188829844s) [seg_pos={0 0}, upload=110132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 13 03:14:31.757833714 (105h48m26.188913845s) [upload=110132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 13 03:14:31.757837731 (105h48m26.188917862s) [upload=110132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:14:32.059045060 (105h48m26.490125191s) [upload=110132]: Done uploading segments... Aug 13 03:14:32.059061040 (105h48m26.490141171s) [upload=110132]: Upload finished. err: <nil> Duration: 6.133673063s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 1.981581456s FAIL (2025-08-13 03:25:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 13 03:23:34.987469695 (105h57m29.418549826s) [upload=110284]: Starting upload Aug 13 03:23:34.987520002 (105h57m29.418600133s) [upload=110284]: Done starting upload Aug 13 03:23:34.987570067 (105h57m29.418650198s) [upload=110284]: Uploading segments... Aug 13 03:23:34.987604243 (105h57m29.418684374s) [upload=110284]: Got next segment. Inline: true Aug 13 03:23:34.987608641 (105h57m29.418688772s) [upload=110284]: Waiting for error group managing segments... Aug 13 03:23:34.987640973 (105h57m29.418721094s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:23:34.987644710 (105h57m29.418724841s) [upload=110284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:23:34.987647044 (105h57m29.418727165s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:23:35.199946062 (105h57m29.631026193s) [upload=110284]: Done uploading segments... Aug 13 03:23:35.199964076 (105h57m29.631044207s) [upload=110284]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.981602946s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X..X.......X...........X...............X................] duration: 5.67593909s FAIL (2025-08-13 03:25:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 07a64daf5ce9db32d41aea6196fcc70ddc5670c6fb7b746f966c0a14f0f90cad != before e0a47ea874bf973a2cf983e7ff42ad385b8577ce8b4f3c6f68efa1aa67839818 ======LOGS====== Aug 13 03:14:27.200063120 (105h48m21.631143251s) [upload=110132]: Starting upload Aug 13 03:14:27.200107645 (105h48m21.631187776s) [upload=110132]: Done starting upload Aug 13 03:14:27.200237103 (105h48m21.631317234s) [upload=110132]: Uploading segments... Aug 13 03:14:27.200364997 (105h48m21.631445128s) [upload=110132]: Got next segment. Inline: false Aug 13 03:14:27.200415604 (105h48m21.631495785s) [upload=110132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:14:27.200424100 (105h48m21.631504271s) [upload=110132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 13 03:14:27.660811382 (105h48m22.091891513s) [seg_pos={0 0}, upload=110132]: Begin upload segment... Aug 13 03:14:27.660865877 (105h48m22.091946008s) [seg_pos={0 0}, upload=110132]: Done begin upload segment. Aug 13 03:14:27.660886587 (105h48m22.091966718s) [noise=true , address=77.90.31.10:4528 , piece_id=XKSFJ2I7HYD72TOTWFHDRRLTWE6SFSMKTDVKYIYFWI6VYQF4EW5Q, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.660900844 (105h48m22.091981055s) [noise=true , address=79.127.219.38:4035 , piece_id=MXSRDYRUPY6DSE6N66WNQZPCC6KVBHISPOIZBGFAQSD3CUGSFWOA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.661384269 (105h48m22.0924644s) [noise=true , address=104.36.82.58:4284 , piece_id=K5J626LGT3TOFW6Q4G22RJJYQGXPYS2B74RB34HVPD6HTOQA2QCA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.661598819 (105h48m22.09267894s) [noise=true , address=104.36.82.14:4250 , piece_id=CQY4HBMBDYVUKR4637GSB33VW6G4OEP53VH27DMSBPS54CATSBWA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.661669895 (105h48m22.092750026s) [noise=true , address=104.36.82.47:4544 , piece_id=TIKP5LMX65MWSBUUPMX36JYRVFC3JVCA5KLRS7CIZMF3SJVS73OA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.661996209 (105h48m22.09307634s) [noise=true , address=77.90.31.14:4679 , piece_id=CFPI32OHVSZTT2YEOSL5ZXAHA37PT57VMTTTCTTIRQVMXAJM6D5A, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.662169181 (105h48m22.093249342s) [noise=true , address=77.90.31.10:4530 , piece_id=JILOKHNOQCGIDXTL2NQF4HCNYE6RJXNH4OICDIVSHADVBIUVA4ZA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.662275895 (105h48m22.093356016s) [noise=true , address=104.36.82.18:4377 , piece_id=Y7NIH46TSTZBIMNAK6WLRFX6WYAZR2BPQLVKHJ4OK44TTD3JP2AA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.662522045 (105h48m22.093602166s) [noise=true , address=104.36.82.9:4024 , piece_id=Y5CY7VLCSC6BNGIH6NF6446BOU4LNIX5GCCY6DSWCPKXPCK4QDQQ, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.662779709 (105h48m22.09385984s) [noise=true , address=104.36.82.46:4484 , piece_id=YRE57GHVKPCOHVRDGSL4FGGX4NPXUJF4L7ELHZGHJ4EC4QDMEU3A, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.662985592 (105h48m22.094065723s) [noise=true , address=104.36.82.59:4312 , piece_id=FZB6SMU32MTZTBAHUTHLWVZGQ53C67FAAIIH6VE2MTLS6PGFXXZQ, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.663176938 (105h48m22.094257069s) [noise=true , address=104.36.82.60:4352 , piece_id=UCDAKY7PVPM2LXMCEZYWZNQJV6DJRZIBB2VMJP2IB6N4ZEU6Q7IA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.663367453 (105h48m22.094447584s) [noise=true , address=104.36.82.16:4299 , piece_id=CTNAIZNCPYLZLJZSH7OVSKBIJ2Q3J2QC75ONHCNPKVGRQEQWLL2Q, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.663568779 (105h48m22.09464891s) [noise=true , address=104.36.82.37:4678 , piece_id=6JTI7SDS4DTDJFAF4FZCCCOON7KOPOXXT5MZ2MDIPSQXZZKF6TKQ, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.663767839 (105h48m22.09484797s) [noise=true , address=104.36.82.36:4637 , piece_id=QXP45IJF7253J7A5MY3C4DAA4VFFT5XULJ3Y4QTXTDCTVJP4IYGA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.663960167 (105h48m22.095040288s) [noise=true , address=104.36.82.23:4472 , piece_id=IRJTNTCAIGVWU4HIE3L3ZCZH6VM6FHFU3SEZDXFYEV4R2BAP45RA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.664144580 (105h48m22.095224711s) [noise=true , address=104.36.82.17:4329 , piece_id=KPGYZK5QIGRNMUHQLFRGRNIPI5EONZBHCGARRSDZCK2UFJLNN4TA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.664326608 (105h48m22.095406739s) [noise=true , address=104.36.82.11:4138 , piece_id=7L7M5DMAFNHOHVMUAQPFOKLU4TKIBL3TMS6D2F6X5T23BKDKGS7Q, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:29.202132828 (105h48m23.633212949s) [upload=110132]: Next returned nil segment Aug 13 03:14:29.202151514 (105h48m23.633231645s) [upload=110132]: Waiting for error group managing segments... Aug 13 03:14:29.624905241 (105h48m24.055985372s) [noise=true , address=77.90.31.14:4679 , piece_id=CFPI32OHVSZTT2YEOSL5ZXAHA37PT57VMTTTCTTIRQVMXAJM6D5A, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.625120442 (105h48m24.056200574s) [noise=true , address=104.36.82.59:4312 , piece_id=FZB6SMU32MTZTBAHUTHLWVZGQ53C67FAAIIH6VE2MTLS6PGFXXZQ, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.667080856 (105h48m24.098160987s) [noise=true , address=104.36.82.9:4024 , piece_id=Y5CY7VLCSC6BNGIH6NF6446BOU4LNIX5GCCY6DSWCPKXPCK4QDQQ, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.669032951 (105h48m24.100113082s) [noise=true , address=104.36.82.14:4250 , piece_id=CQY4HBMBDYVUKR4637GSB33VW6G4OEP53VH27DMSBPS54CATSBWA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.699132833 (105h48m24.130212964s) [noise=true , address=104.36.82.11:4138 , piece_id=7L7M5DMAFNHOHVMUAQPFOKLU4TKIBL3TMS6D2F6X5T23BKDKGS7Q, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.717982878 (105h48m24.149063009s) [noise=true , address=104.36.82.36:4637 , piece_id=QXP45IJF7253J7A5MY3C4DAA4VFFT5XULJ3Y4QTXTDCTVJP4IYGA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.724446195 (105h48m24.155526316s) [noise=true , address=104.36.82.37:4678 , piece_id=6JTI7SDS4DTDJFAF4FZCCCOON7KOPOXXT5MZ2MDIPSQXZZKF6TKQ, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.731783564 (105h48m24.162863695s) [noise=true , address=104.36.82.58:4284 , piece_id=K5J626LGT3TOFW6Q4G22RJJYQGXPYS2B74RB34HVPD6HTOQA2QCA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.780951378 (105h48m24.212031509s) [noise=true , address=104.36.82.46:4484 , piece_id=YRE57GHVKPCOHVRDGSL4FGGX4NPXUJF4L7ELHZGHJ4EC4QDMEU3A, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.794319577 (105h48m24.225399708s) [noise=true , address=104.36.82.47:4544 , piece_id=TIKP5LMX65MWSBUUPMX36JYRVFC3JVCA5KLRS7CIZMF3SJVS73OA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:30.252036350 (105h48m24.683116481s) [noise=true , address=79.127.219.38:4035 , piece_id=MXSRDYRUPY6DSE6N66WNQZPCC6KVBHISPOIZBGFAQSD3CUGSFWOA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:30.523524634 (105h48m24.954604765s) [noise=true , address=104.36.82.23:4472 , piece_id=IRJTNTCAIGVWU4HIE3L3ZCZH6VM6FHFU3SEZDXFYEV4R2BAP45RA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:31.207986103 (105h48m25.639066235s) [noise=true , address=77.90.31.10:4528 , piece_id=XKSFJ2I7HYD72TOTWFHDRRLTWE6SFSMKTDVKYIYFWI6VYQF4EW5Q, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:31.757453767 (105h48m26.188533898s) [noise=true , address=104.36.82.16:4299 , piece_id=CTNAIZNCPYLZLJZSH7OVSKBIJ2Q3J2QC75ONHCNPKVGRQEQWLL2Q, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:31.757477813 (105h48m26.188557934s) [seg_pos={0 0}, upload=110132]: Segment reached optimal threshold of 14 pieces. Aug 13 03:14:31.757561022 (105h48m26.188641153s) [noise=true , address=77.90.31.10:4530 , piece_id=JILOKHNOQCGIDXTL2NQF4HCNYE6RJXNH4OICDIVSHADVBIUVA4ZA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne): protocol: expected piece hash; context canceled; EOF Aug 13 03:14:31.757627769 (105h48m26.1887079s) [noise=true , address=104.36.82.18:4377 , piece_id=Y7NIH46TSTZBIMNAK6WLRFX6WYAZR2BPQLVKHJ4OK44TTD3JP2AA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JHy4o8dxFNeeiitLPavxspcf4rL9KfckcRAuC121v8YZG6qmT): protocol: expected piece hash; context canceled; EOF Aug 13 03:14:31.757682684 (105h48m26.188762815s) [noise=true , address=104.36.82.17:4329 , piece_id=KPGYZK5QIGRNMUHQLFRGRNIPI5EONZBHCGARRSDZCK2UFJLNN4TA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: expected piece hash; context canceled; EOF Aug 13 03:14:31.757703594 (105h48m26.188783725s) [noise=true , address=104.36.82.60:4352 , piece_id=UCDAKY7PVPM2LXMCEZYWZNQJV6DJRZIBB2VMJP2IB6N4ZEU6Q7IA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: write tcp 188.166.225.170:47910->104.36.82.60:4352: use of closed network connection; write tcp 188.166.225.170:47910->104.36.82.60:4352: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47910->104.36.82.60:4352: use of closed network connection Aug 13 03:14:31.757749713 (105h48m26.188829844s) [seg_pos={0 0}, upload=110132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 13 03:14:31.757833714 (105h48m26.188913845s) [upload=110132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 13 03:14:31.757837731 (105h48m26.188917862s) [upload=110132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:14:32.059045060 (105h48m26.490125191s) [upload=110132]: Done uploading segments... Aug 13 03:14:32.059061040 (105h48m26.490141171s) [upload=110132]: Upload finished. err: <nil> Duration: 5.675965602s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 1.932018974s FAIL (2025-08-13 03:25:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 13 03:06:41.898033585 (105h40m36.329113716s) [upload=110007]: Starting upload Aug 13 03:06:41.898093971 (105h40m36.329174102s) [upload=110007]: Done starting upload Aug 13 03:06:41.898140650 (105h40m36.329220781s) [upload=110007]: Uploading segments... Aug 13 03:06:41.898172331 (105h40m36.329252462s) [upload=110007]: Got next segment. Inline: true Aug 13 03:06:41.898176078 (105h40m36.329256209s) [upload=110007]: Waiting for error group managing segments... Aug 13 03:06:41.898208831 (105h40m36.329288962s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:06:41.898212778 (105h40m36.329292909s) [upload=110007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:06:41.898215563 (105h40m36.329295694s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:07:11.244546166 (105h41m5.675626307s) [upload=110007]: Done uploading segments... Aug 13 03:07:11.244585330 (105h41m5.675665482s) [upload=110007]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.932045467s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................................X.....XX.......] duration: 15.011154069s FAIL (2025-08-13 03:25:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.011178084s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 1.949792713s FAIL (2025-08-13 03:24:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 13 03:23:34.987469695 (105h57m29.418549826s) [upload=110284]: Starting upload Aug 13 03:23:34.987520002 (105h57m29.418600133s) [upload=110284]: Done starting upload Aug 13 03:23:34.987570067 (105h57m29.418650198s) [upload=110284]: Uploading segments... Aug 13 03:23:34.987604243 (105h57m29.418684374s) [upload=110284]: Got next segment. Inline: true Aug 13 03:23:34.987608641 (105h57m29.418688772s) [upload=110284]: Waiting for error group managing segments... Aug 13 03:23:34.987640973 (105h57m29.418721094s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:23:34.987644710 (105h57m29.418724841s) [upload=110284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:23:34.987647044 (105h57m29.418727165s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:23:35.199946062 (105h57m29.631026193s) [upload=110284]: Done uploading segments... Aug 13 03:23:35.199964076 (105h57m29.631044207s) [upload=110284]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.949810376s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 1.897184018s FAIL (2025-08-13 03:24:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 13 03:06:41.898033585 (105h40m36.329113716s) [upload=110007]: Starting upload Aug 13 03:06:41.898093971 (105h40m36.329174102s) [upload=110007]: Done starting upload Aug 13 03:06:41.898140650 (105h40m36.329220781s) [upload=110007]: Uploading segments... Aug 13 03:06:41.898172331 (105h40m36.329252462s) [upload=110007]: Got next segment. Inline: true Aug 13 03:06:41.898176078 (105h40m36.329256209s) [upload=110007]: Waiting for error group managing segments... Aug 13 03:06:41.898208831 (105h40m36.329288962s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:06:41.898212778 (105h40m36.329292909s) [upload=110007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:06:41.898215563 (105h40m36.329295694s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:07:11.244546166 (105h41m5.675626307s) [upload=110007]: Done uploading segments... Aug 13 03:07:11.244585330 (105h41m5.675665482s) [upload=110007]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.897218324s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X..X.......X...........X...............X...............] duration: 5.980249942s FAIL (2025-08-13 03:24:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 07a64daf5ce9db32d41aea6196fcc70ddc5670c6fb7b746f966c0a14f0f90cad != before e0a47ea874bf973a2cf983e7ff42ad385b8577ce8b4f3c6f68efa1aa67839818 ======LOGS====== Aug 13 03:14:27.200063120 (105h48m21.631143251s) [upload=110132]: Starting upload Aug 13 03:14:27.200107645 (105h48m21.631187776s) [upload=110132]: Done starting upload Aug 13 03:14:27.200237103 (105h48m21.631317234s) [upload=110132]: Uploading segments... Aug 13 03:14:27.200364997 (105h48m21.631445128s) [upload=110132]: Got next segment. Inline: false Aug 13 03:14:27.200415604 (105h48m21.631495785s) [upload=110132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:14:27.200424100 (105h48m21.631504271s) [upload=110132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 13 03:14:27.660811382 (105h48m22.091891513s) [seg_pos={0 0}, upload=110132]: Begin upload segment... Aug 13 03:14:27.660865877 (105h48m22.091946008s) [seg_pos={0 0}, upload=110132]: Done begin upload segment. Aug 13 03:14:27.660886587 (105h48m22.091966718s) [noise=true , address=77.90.31.10:4528 , piece_id=XKSFJ2I7HYD72TOTWFHDRRLTWE6SFSMKTDVKYIYFWI6VYQF4EW5Q, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.660900844 (105h48m22.091981055s) [noise=true , address=79.127.219.38:4035 , piece_id=MXSRDYRUPY6DSE6N66WNQZPCC6KVBHISPOIZBGFAQSD3CUGSFWOA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.661384269 (105h48m22.0924644s) [noise=true , address=104.36.82.58:4284 , piece_id=K5J626LGT3TOFW6Q4G22RJJYQGXPYS2B74RB34HVPD6HTOQA2QCA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.661598819 (105h48m22.09267894s) [noise=true , address=104.36.82.14:4250 , piece_id=CQY4HBMBDYVUKR4637GSB33VW6G4OEP53VH27DMSBPS54CATSBWA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.661669895 (105h48m22.092750026s) [noise=true , address=104.36.82.47:4544 , piece_id=TIKP5LMX65MWSBUUPMX36JYRVFC3JVCA5KLRS7CIZMF3SJVS73OA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.661996209 (105h48m22.09307634s) [noise=true , address=77.90.31.14:4679 , piece_id=CFPI32OHVSZTT2YEOSL5ZXAHA37PT57VMTTTCTTIRQVMXAJM6D5A, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.662169181 (105h48m22.093249342s) [noise=true , address=77.90.31.10:4530 , piece_id=JILOKHNOQCGIDXTL2NQF4HCNYE6RJXNH4OICDIVSHADVBIUVA4ZA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.662275895 (105h48m22.093356016s) [noise=true , address=104.36.82.18:4377 , piece_id=Y7NIH46TSTZBIMNAK6WLRFX6WYAZR2BPQLVKHJ4OK44TTD3JP2AA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.662522045 (105h48m22.093602166s) [noise=true , address=104.36.82.9:4024 , piece_id=Y5CY7VLCSC6BNGIH6NF6446BOU4LNIX5GCCY6DSWCPKXPCK4QDQQ, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.662779709 (105h48m22.09385984s) [noise=true , address=104.36.82.46:4484 , piece_id=YRE57GHVKPCOHVRDGSL4FGGX4NPXUJF4L7ELHZGHJ4EC4QDMEU3A, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.662985592 (105h48m22.094065723s) [noise=true , address=104.36.82.59:4312 , piece_id=FZB6SMU32MTZTBAHUTHLWVZGQ53C67FAAIIH6VE2MTLS6PGFXXZQ, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.663176938 (105h48m22.094257069s) [noise=true , address=104.36.82.60:4352 , piece_id=UCDAKY7PVPM2LXMCEZYWZNQJV6DJRZIBB2VMJP2IB6N4ZEU6Q7IA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.663367453 (105h48m22.094447584s) [noise=true , address=104.36.82.16:4299 , piece_id=CTNAIZNCPYLZLJZSH7OVSKBIJ2Q3J2QC75ONHCNPKVGRQEQWLL2Q, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.663568779 (105h48m22.09464891s) [noise=true , address=104.36.82.37:4678 , piece_id=6JTI7SDS4DTDJFAF4FZCCCOON7KOPOXXT5MZ2MDIPSQXZZKF6TKQ, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.663767839 (105h48m22.09484797s) [noise=true , address=104.36.82.36:4637 , piece_id=QXP45IJF7253J7A5MY3C4DAA4VFFT5XULJ3Y4QTXTDCTVJP4IYGA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.663960167 (105h48m22.095040288s) [noise=true , address=104.36.82.23:4472 , piece_id=IRJTNTCAIGVWU4HIE3L3ZCZH6VM6FHFU3SEZDXFYEV4R2BAP45RA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.664144580 (105h48m22.095224711s) [noise=true , address=104.36.82.17:4329 , piece_id=KPGYZK5QIGRNMUHQLFRGRNIPI5EONZBHCGARRSDZCK2UFJLNN4TA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.664326608 (105h48m22.095406739s) [noise=true , address=104.36.82.11:4138 , piece_id=7L7M5DMAFNHOHVMUAQPFOKLU4TKIBL3TMS6D2F6X5T23BKDKGS7Q, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:29.202132828 (105h48m23.633212949s) [upload=110132]: Next returned nil segment Aug 13 03:14:29.202151514 (105h48m23.633231645s) [upload=110132]: Waiting for error group managing segments... Aug 13 03:14:29.624905241 (105h48m24.055985372s) [noise=true , address=77.90.31.14:4679 , piece_id=CFPI32OHVSZTT2YEOSL5ZXAHA37PT57VMTTTCTTIRQVMXAJM6D5A, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.625120442 (105h48m24.056200574s) [noise=true , address=104.36.82.59:4312 , piece_id=FZB6SMU32MTZTBAHUTHLWVZGQ53C67FAAIIH6VE2MTLS6PGFXXZQ, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.667080856 (105h48m24.098160987s) [noise=true , address=104.36.82.9:4024 , piece_id=Y5CY7VLCSC6BNGIH6NF6446BOU4LNIX5GCCY6DSWCPKXPCK4QDQQ, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.669032951 (105h48m24.100113082s) [noise=true , address=104.36.82.14:4250 , piece_id=CQY4HBMBDYVUKR4637GSB33VW6G4OEP53VH27DMSBPS54CATSBWA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.699132833 (105h48m24.130212964s) [noise=true , address=104.36.82.11:4138 , piece_id=7L7M5DMAFNHOHVMUAQPFOKLU4TKIBL3TMS6D2F6X5T23BKDKGS7Q, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.717982878 (105h48m24.149063009s) [noise=true , address=104.36.82.36:4637 , piece_id=QXP45IJF7253J7A5MY3C4DAA4VFFT5XULJ3Y4QTXTDCTVJP4IYGA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.724446195 (105h48m24.155526316s) [noise=true , address=104.36.82.37:4678 , piece_id=6JTI7SDS4DTDJFAF4FZCCCOON7KOPOXXT5MZ2MDIPSQXZZKF6TKQ, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.731783564 (105h48m24.162863695s) [noise=true , address=104.36.82.58:4284 , piece_id=K5J626LGT3TOFW6Q4G22RJJYQGXPYS2B74RB34HVPD6HTOQA2QCA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.780951378 (105h48m24.212031509s) [noise=true , address=104.36.82.46:4484 , piece_id=YRE57GHVKPCOHVRDGSL4FGGX4NPXUJF4L7ELHZGHJ4EC4QDMEU3A, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.794319577 (105h48m24.225399708s) [noise=true , address=104.36.82.47:4544 , piece_id=TIKP5LMX65MWSBUUPMX36JYRVFC3JVCA5KLRS7CIZMF3SJVS73OA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:30.252036350 (105h48m24.683116481s) [noise=true , address=79.127.219.38:4035 , piece_id=MXSRDYRUPY6DSE6N66WNQZPCC6KVBHISPOIZBGFAQSD3CUGSFWOA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:30.523524634 (105h48m24.954604765s) [noise=true , address=104.36.82.23:4472 , piece_id=IRJTNTCAIGVWU4HIE3L3ZCZH6VM6FHFU3SEZDXFYEV4R2BAP45RA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:31.207986103 (105h48m25.639066235s) [noise=true , address=77.90.31.10:4528 , piece_id=XKSFJ2I7HYD72TOTWFHDRRLTWE6SFSMKTDVKYIYFWI6VYQF4EW5Q, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:31.757453767 (105h48m26.188533898s) [noise=true , address=104.36.82.16:4299 , piece_id=CTNAIZNCPYLZLJZSH7OVSKBIJ2Q3J2QC75ONHCNPKVGRQEQWLL2Q, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:31.757477813 (105h48m26.188557934s) [seg_pos={0 0}, upload=110132]: Segment reached optimal threshold of 14 pieces. Aug 13 03:14:31.757561022 (105h48m26.188641153s) [noise=true , address=77.90.31.10:4530 , piece_id=JILOKHNOQCGIDXTL2NQF4HCNYE6RJXNH4OICDIVSHADVBIUVA4ZA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne): protocol: expected piece hash; context canceled; EOF Aug 13 03:14:31.757627769 (105h48m26.1887079s) [noise=true , address=104.36.82.18:4377 , piece_id=Y7NIH46TSTZBIMNAK6WLRFX6WYAZR2BPQLVKHJ4OK44TTD3JP2AA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JHy4o8dxFNeeiitLPavxspcf4rL9KfckcRAuC121v8YZG6qmT): protocol: expected piece hash; context canceled; EOF Aug 13 03:14:31.757682684 (105h48m26.188762815s) [noise=true , address=104.36.82.17:4329 , piece_id=KPGYZK5QIGRNMUHQLFRGRNIPI5EONZBHCGARRSDZCK2UFJLNN4TA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: expected piece hash; context canceled; EOF Aug 13 03:14:31.757703594 (105h48m26.188783725s) [noise=true , address=104.36.82.60:4352 , piece_id=UCDAKY7PVPM2LXMCEZYWZNQJV6DJRZIBB2VMJP2IB6N4ZEU6Q7IA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: write tcp 188.166.225.170:47910->104.36.82.60:4352: use of closed network connection; write tcp 188.166.225.170:47910->104.36.82.60:4352: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47910->104.36.82.60:4352: use of closed network connection Aug 13 03:14:31.757749713 (105h48m26.188829844s) [seg_pos={0 0}, upload=110132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 13 03:14:31.757833714 (105h48m26.188913845s) [upload=110132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 13 03:14:31.757837731 (105h48m26.188917862s) [upload=110132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:14:32.059045060 (105h48m26.490125191s) [upload=110132]: Done uploading segments... Aug 13 03:14:32.059061040 (105h48m26.490141171s) [upload=110132]: Upload finished. err: <nil> Duration: 5.980275229s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 1.887303319s FAIL (2025-08-13 03:24:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 13 03:06:41.898033585 (105h40m36.329113716s) [upload=110007]: Starting upload Aug 13 03:06:41.898093971 (105h40m36.329174102s) [upload=110007]: Done starting upload Aug 13 03:06:41.898140650 (105h40m36.329220781s) [upload=110007]: Uploading segments... Aug 13 03:06:41.898172331 (105h40m36.329252462s) [upload=110007]: Got next segment. Inline: true Aug 13 03:06:41.898176078 (105h40m36.329256209s) [upload=110007]: Waiting for error group managing segments... Aug 13 03:06:41.898208831 (105h40m36.329288962s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:06:41.898212778 (105h40m36.329292909s) [upload=110007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:06:41.898215563 (105h40m36.329295694s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:07:11.244546166 (105h41m5.675626307s) [upload=110007]: Done uploading segments... Aug 13 03:07:11.244585330 (105h41m5.675665482s) [upload=110007]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.887327145s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 1.961495032s FAIL (2025-08-13 03:24:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 13 03:23:34.987469695 (105h57m29.418549826s) [upload=110284]: Starting upload Aug 13 03:23:34.987520002 (105h57m29.418600133s) [upload=110284]: Done starting upload Aug 13 03:23:34.987570067 (105h57m29.418650198s) [upload=110284]: Uploading segments... Aug 13 03:23:34.987604243 (105h57m29.418684374s) [upload=110284]: Got next segment. Inline: true Aug 13 03:23:34.987608641 (105h57m29.418688772s) [upload=110284]: Waiting for error group managing segments... Aug 13 03:23:34.987640973 (105h57m29.418721094s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:23:34.987644710 (105h57m29.418724841s) [upload=110284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:23:34.987647044 (105h57m29.418727165s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:23:35.199946062 (105h57m29.631026193s) [upload=110284]: Done uploading segments... Aug 13 03:23:35.199964076 (105h57m29.631044207s) [upload=110284]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.961521892s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................................X.....XX......] duration: 7.269746273s FAIL (2025-08-13 03:23:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.269770549s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X..X.......X...........X...............X..............] duration: 5.929913372s FAIL (2025-08-13 03:23:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 07a64daf5ce9db32d41aea6196fcc70ddc5670c6fb7b746f966c0a14f0f90cad != before e0a47ea874bf973a2cf983e7ff42ad385b8577ce8b4f3c6f68efa1aa67839818 ======LOGS====== Aug 13 03:14:27.200063120 (105h48m21.631143251s) [upload=110132]: Starting upload Aug 13 03:14:27.200107645 (105h48m21.631187776s) [upload=110132]: Done starting upload Aug 13 03:14:27.200237103 (105h48m21.631317234s) [upload=110132]: Uploading segments... Aug 13 03:14:27.200364997 (105h48m21.631445128s) [upload=110132]: Got next segment. Inline: false Aug 13 03:14:27.200415604 (105h48m21.631495785s) [upload=110132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:14:27.200424100 (105h48m21.631504271s) [upload=110132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 13 03:14:27.660811382 (105h48m22.091891513s) [seg_pos={0 0}, upload=110132]: Begin upload segment... Aug 13 03:14:27.660865877 (105h48m22.091946008s) [seg_pos={0 0}, upload=110132]: Done begin upload segment. Aug 13 03:14:27.660886587 (105h48m22.091966718s) [noise=true , address=77.90.31.10:4528 , piece_id=XKSFJ2I7HYD72TOTWFHDRRLTWE6SFSMKTDVKYIYFWI6VYQF4EW5Q, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.660900844 (105h48m22.091981055s) [noise=true , address=79.127.219.38:4035 , piece_id=MXSRDYRUPY6DSE6N66WNQZPCC6KVBHISPOIZBGFAQSD3CUGSFWOA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.661384269 (105h48m22.0924644s) [noise=true , address=104.36.82.58:4284 , piece_id=K5J626LGT3TOFW6Q4G22RJJYQGXPYS2B74RB34HVPD6HTOQA2QCA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.661598819 (105h48m22.09267894s) [noise=true , address=104.36.82.14:4250 , piece_id=CQY4HBMBDYVUKR4637GSB33VW6G4OEP53VH27DMSBPS54CATSBWA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.661669895 (105h48m22.092750026s) [noise=true , address=104.36.82.47:4544 , piece_id=TIKP5LMX65MWSBUUPMX36JYRVFC3JVCA5KLRS7CIZMF3SJVS73OA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.661996209 (105h48m22.09307634s) [noise=true , address=77.90.31.14:4679 , piece_id=CFPI32OHVSZTT2YEOSL5ZXAHA37PT57VMTTTCTTIRQVMXAJM6D5A, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.662169181 (105h48m22.093249342s) [noise=true , address=77.90.31.10:4530 , piece_id=JILOKHNOQCGIDXTL2NQF4HCNYE6RJXNH4OICDIVSHADVBIUVA4ZA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.662275895 (105h48m22.093356016s) [noise=true , address=104.36.82.18:4377 , piece_id=Y7NIH46TSTZBIMNAK6WLRFX6WYAZR2BPQLVKHJ4OK44TTD3JP2AA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.662522045 (105h48m22.093602166s) [noise=true , address=104.36.82.9:4024 , piece_id=Y5CY7VLCSC6BNGIH6NF6446BOU4LNIX5GCCY6DSWCPKXPCK4QDQQ, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.662779709 (105h48m22.09385984s) [noise=true , address=104.36.82.46:4484 , piece_id=YRE57GHVKPCOHVRDGSL4FGGX4NPXUJF4L7ELHZGHJ4EC4QDMEU3A, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.662985592 (105h48m22.094065723s) [noise=true , address=104.36.82.59:4312 , piece_id=FZB6SMU32MTZTBAHUTHLWVZGQ53C67FAAIIH6VE2MTLS6PGFXXZQ, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.663176938 (105h48m22.094257069s) [noise=true , address=104.36.82.60:4352 , piece_id=UCDAKY7PVPM2LXMCEZYWZNQJV6DJRZIBB2VMJP2IB6N4ZEU6Q7IA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.663367453 (105h48m22.094447584s) [noise=true , address=104.36.82.16:4299 , piece_id=CTNAIZNCPYLZLJZSH7OVSKBIJ2Q3J2QC75ONHCNPKVGRQEQWLL2Q, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.663568779 (105h48m22.09464891s) [noise=true , address=104.36.82.37:4678 , piece_id=6JTI7SDS4DTDJFAF4FZCCCOON7KOPOXXT5MZ2MDIPSQXZZKF6TKQ, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.663767839 (105h48m22.09484797s) [noise=true , address=104.36.82.36:4637 , piece_id=QXP45IJF7253J7A5MY3C4DAA4VFFT5XULJ3Y4QTXTDCTVJP4IYGA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.663960167 (105h48m22.095040288s) [noise=true , address=104.36.82.23:4472 , piece_id=IRJTNTCAIGVWU4HIE3L3ZCZH6VM6FHFU3SEZDXFYEV4R2BAP45RA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.664144580 (105h48m22.095224711s) [noise=true , address=104.36.82.17:4329 , piece_id=KPGYZK5QIGRNMUHQLFRGRNIPI5EONZBHCGARRSDZCK2UFJLNN4TA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.664326608 (105h48m22.095406739s) [noise=true , address=104.36.82.11:4138 , piece_id=7L7M5DMAFNHOHVMUAQPFOKLU4TKIBL3TMS6D2F6X5T23BKDKGS7Q, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:29.202132828 (105h48m23.633212949s) [upload=110132]: Next returned nil segment Aug 13 03:14:29.202151514 (105h48m23.633231645s) [upload=110132]: Waiting for error group managing segments... Aug 13 03:14:29.624905241 (105h48m24.055985372s) [noise=true , address=77.90.31.14:4679 , piece_id=CFPI32OHVSZTT2YEOSL5ZXAHA37PT57VMTTTCTTIRQVMXAJM6D5A, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.625120442 (105h48m24.056200574s) [noise=true , address=104.36.82.59:4312 , piece_id=FZB6SMU32MTZTBAHUTHLWVZGQ53C67FAAIIH6VE2MTLS6PGFXXZQ, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.667080856 (105h48m24.098160987s) [noise=true , address=104.36.82.9:4024 , piece_id=Y5CY7VLCSC6BNGIH6NF6446BOU4LNIX5GCCY6DSWCPKXPCK4QDQQ, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.669032951 (105h48m24.100113082s) [noise=true , address=104.36.82.14:4250 , piece_id=CQY4HBMBDYVUKR4637GSB33VW6G4OEP53VH27DMSBPS54CATSBWA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.699132833 (105h48m24.130212964s) [noise=true , address=104.36.82.11:4138 , piece_id=7L7M5DMAFNHOHVMUAQPFOKLU4TKIBL3TMS6D2F6X5T23BKDKGS7Q, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.717982878 (105h48m24.149063009s) [noise=true , address=104.36.82.36:4637 , piece_id=QXP45IJF7253J7A5MY3C4DAA4VFFT5XULJ3Y4QTXTDCTVJP4IYGA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.724446195 (105h48m24.155526316s) [noise=true , address=104.36.82.37:4678 , piece_id=6JTI7SDS4DTDJFAF4FZCCCOON7KOPOXXT5MZ2MDIPSQXZZKF6TKQ, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.731783564 (105h48m24.162863695s) [noise=true , address=104.36.82.58:4284 , piece_id=K5J626LGT3TOFW6Q4G22RJJYQGXPYS2B74RB34HVPD6HTOQA2QCA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.780951378 (105h48m24.212031509s) [noise=true , address=104.36.82.46:4484 , piece_id=YRE57GHVKPCOHVRDGSL4FGGX4NPXUJF4L7ELHZGHJ4EC4QDMEU3A, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.794319577 (105h48m24.225399708s) [noise=true , address=104.36.82.47:4544 , piece_id=TIKP5LMX65MWSBUUPMX36JYRVFC3JVCA5KLRS7CIZMF3SJVS73OA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:30.252036350 (105h48m24.683116481s) [noise=true , address=79.127.219.38:4035 , piece_id=MXSRDYRUPY6DSE6N66WNQZPCC6KVBHISPOIZBGFAQSD3CUGSFWOA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:30.523524634 (105h48m24.954604765s) [noise=true , address=104.36.82.23:4472 , piece_id=IRJTNTCAIGVWU4HIE3L3ZCZH6VM6FHFU3SEZDXFYEV4R2BAP45RA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:31.207986103 (105h48m25.639066235s) [noise=true , address=77.90.31.10:4528 , piece_id=XKSFJ2I7HYD72TOTWFHDRRLTWE6SFSMKTDVKYIYFWI6VYQF4EW5Q, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:31.757453767 (105h48m26.188533898s) [noise=true , address=104.36.82.16:4299 , piece_id=CTNAIZNCPYLZLJZSH7OVSKBIJ2Q3J2QC75ONHCNPKVGRQEQWLL2Q, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:31.757477813 (105h48m26.188557934s) [seg_pos={0 0}, upload=110132]: Segment reached optimal threshold of 14 pieces. Aug 13 03:14:31.757561022 (105h48m26.188641153s) [noise=true , address=77.90.31.10:4530 , piece_id=JILOKHNOQCGIDXTL2NQF4HCNYE6RJXNH4OICDIVSHADVBIUVA4ZA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne): protocol: expected piece hash; context canceled; EOF Aug 13 03:14:31.757627769 (105h48m26.1887079s) [noise=true , address=104.36.82.18:4377 , piece_id=Y7NIH46TSTZBIMNAK6WLRFX6WYAZR2BPQLVKHJ4OK44TTD3JP2AA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JHy4o8dxFNeeiitLPavxspcf4rL9KfckcRAuC121v8YZG6qmT): protocol: expected piece hash; context canceled; EOF Aug 13 03:14:31.757682684 (105h48m26.188762815s) [noise=true , address=104.36.82.17:4329 , piece_id=KPGYZK5QIGRNMUHQLFRGRNIPI5EONZBHCGARRSDZCK2UFJLNN4TA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: expected piece hash; context canceled; EOF Aug 13 03:14:31.757703594 (105h48m26.188783725s) [noise=true , address=104.36.82.60:4352 , piece_id=UCDAKY7PVPM2LXMCEZYWZNQJV6DJRZIBB2VMJP2IB6N4ZEU6Q7IA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: write tcp 188.166.225.170:47910->104.36.82.60:4352: use of closed network connection; write tcp 188.166.225.170:47910->104.36.82.60:4352: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47910->104.36.82.60:4352: use of closed network connection Aug 13 03:14:31.757749713 (105h48m26.188829844s) [seg_pos={0 0}, upload=110132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 13 03:14:31.757833714 (105h48m26.188913845s) [upload=110132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 13 03:14:31.757837731 (105h48m26.188917862s) [upload=110132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:14:32.059045060 (105h48m26.490125191s) [upload=110132]: Done uploading segments... Aug 13 03:14:32.059061040 (105h48m26.490141171s) [upload=110132]: Upload finished. err: <nil> Duration: 5.929931617s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 1.90442367s FAIL (2025-08-13 03:23:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 13 03:06:41.898033585 (105h40m36.329113716s) [upload=110007]: Starting upload Aug 13 03:06:41.898093971 (105h40m36.329174102s) [upload=110007]: Done starting upload Aug 13 03:06:41.898140650 (105h40m36.329220781s) [upload=110007]: Uploading segments... Aug 13 03:06:41.898172331 (105h40m36.329252462s) [upload=110007]: Got next segment. Inline: true Aug 13 03:06:41.898176078 (105h40m36.329256209s) [upload=110007]: Waiting for error group managing segments... Aug 13 03:06:41.898208831 (105h40m36.329288962s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:06:41.898212778 (105h40m36.329292909s) [upload=110007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:06:41.898215563 (105h40m36.329295694s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:07:11.244546166 (105h41m5.675626307s) [upload=110007]: Done uploading segments... Aug 13 03:07:11.244585330 (105h41m5.675665482s) [upload=110007]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.904448317s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [X..........................................................X] duration: 855.422065ms FAIL (2025-08-13 03:23:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 13 03:23:34.987469695 (105h57m29.418549826s) [upload=110284]: Starting upload Aug 13 03:23:34.987520002 (105h57m29.418600133s) [upload=110284]: Done starting upload Aug 13 03:23:34.987570067 (105h57m29.418650198s) [upload=110284]: Uploading segments... Aug 13 03:23:34.987604243 (105h57m29.418684374s) [upload=110284]: Got next segment. Inline: true Aug 13 03:23:34.987608641 (105h57m29.418688772s) [upload=110284]: Waiting for error group managing segments... Aug 13 03:23:34.987640973 (105h57m29.418721094s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:23:34.987644710 (105h57m29.418724841s) [upload=110284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:23:34.987647044 (105h57m29.418727165s) [upload=110284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:23:35.199946062 (105h57m29.631026193s) [upload=110284]: Done uploading segments... Aug 13 03:23:35.199964076 (105h57m29.631044207s) [upload=110284]: Upload finished. err: metaclient: Too Many Requests Duration:* 855.438104ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................................X.....XX.....] duration: 7.984546776s FAIL (2025-08-13 03:23:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.984561643s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X..X.......X...........X...............X.............] duration: 5.805187924s FAIL (2025-08-13 03:23:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 07a64daf5ce9db32d41aea6196fcc70ddc5670c6fb7b746f966c0a14f0f90cad != before e0a47ea874bf973a2cf983e7ff42ad385b8577ce8b4f3c6f68efa1aa67839818 ======LOGS====== Aug 13 03:14:27.200063120 (105h48m21.631143251s) [upload=110132]: Starting upload Aug 13 03:14:27.200107645 (105h48m21.631187776s) [upload=110132]: Done starting upload Aug 13 03:14:27.200237103 (105h48m21.631317234s) [upload=110132]: Uploading segments... Aug 13 03:14:27.200364997 (105h48m21.631445128s) [upload=110132]: Got next segment. Inline: false Aug 13 03:14:27.200415604 (105h48m21.631495785s) [upload=110132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:14:27.200424100 (105h48m21.631504271s) [upload=110132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 13 03:14:27.660811382 (105h48m22.091891513s) [seg_pos={0 0}, upload=110132]: Begin upload segment... Aug 13 03:14:27.660865877 (105h48m22.091946008s) [seg_pos={0 0}, upload=110132]: Done begin upload segment. Aug 13 03:14:27.660886587 (105h48m22.091966718s) [noise=true , address=77.90.31.10:4528 , piece_id=XKSFJ2I7HYD72TOTWFHDRRLTWE6SFSMKTDVKYIYFWI6VYQF4EW5Q, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.660900844 (105h48m22.091981055s) [noise=true , address=79.127.219.38:4035 , piece_id=MXSRDYRUPY6DSE6N66WNQZPCC6KVBHISPOIZBGFAQSD3CUGSFWOA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.661384269 (105h48m22.0924644s) [noise=true , address=104.36.82.58:4284 , piece_id=K5J626LGT3TOFW6Q4G22RJJYQGXPYS2B74RB34HVPD6HTOQA2QCA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.661598819 (105h48m22.09267894s) [noise=true , address=104.36.82.14:4250 , piece_id=CQY4HBMBDYVUKR4637GSB33VW6G4OEP53VH27DMSBPS54CATSBWA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.661669895 (105h48m22.092750026s) [noise=true , address=104.36.82.47:4544 , piece_id=TIKP5LMX65MWSBUUPMX36JYRVFC3JVCA5KLRS7CIZMF3SJVS73OA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.661996209 (105h48m22.09307634s) [noise=true , address=77.90.31.14:4679 , piece_id=CFPI32OHVSZTT2YEOSL5ZXAHA37PT57VMTTTCTTIRQVMXAJM6D5A, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.662169181 (105h48m22.093249342s) [noise=true , address=77.90.31.10:4530 , piece_id=JILOKHNOQCGIDXTL2NQF4HCNYE6RJXNH4OICDIVSHADVBIUVA4ZA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.662275895 (105h48m22.093356016s) [noise=true , address=104.36.82.18:4377 , piece_id=Y7NIH46TSTZBIMNAK6WLRFX6WYAZR2BPQLVKHJ4OK44TTD3JP2AA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.662522045 (105h48m22.093602166s) [noise=true , address=104.36.82.9:4024 , piece_id=Y5CY7VLCSC6BNGIH6NF6446BOU4LNIX5GCCY6DSWCPKXPCK4QDQQ, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.662779709 (105h48m22.09385984s) [noise=true , address=104.36.82.46:4484 , piece_id=YRE57GHVKPCOHVRDGSL4FGGX4NPXUJF4L7ELHZGHJ4EC4QDMEU3A, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.662985592 (105h48m22.094065723s) [noise=true , address=104.36.82.59:4312 , piece_id=FZB6SMU32MTZTBAHUTHLWVZGQ53C67FAAIIH6VE2MTLS6PGFXXZQ, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.663176938 (105h48m22.094257069s) [noise=true , address=104.36.82.60:4352 , piece_id=UCDAKY7PVPM2LXMCEZYWZNQJV6DJRZIBB2VMJP2IB6N4ZEU6Q7IA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.663367453 (105h48m22.094447584s) [noise=true , address=104.36.82.16:4299 , piece_id=CTNAIZNCPYLZLJZSH7OVSKBIJ2Q3J2QC75ONHCNPKVGRQEQWLL2Q, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.663568779 (105h48m22.09464891s) [noise=true , address=104.36.82.37:4678 , piece_id=6JTI7SDS4DTDJFAF4FZCCCOON7KOPOXXT5MZ2MDIPSQXZZKF6TKQ, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.663767839 (105h48m22.09484797s) [noise=true , address=104.36.82.36:4637 , piece_id=QXP45IJF7253J7A5MY3C4DAA4VFFT5XULJ3Y4QTXTDCTVJP4IYGA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.663960167 (105h48m22.095040288s) [noise=true , address=104.36.82.23:4472 , piece_id=IRJTNTCAIGVWU4HIE3L3ZCZH6VM6FHFU3SEZDXFYEV4R2BAP45RA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.664144580 (105h48m22.095224711s) [noise=true , address=104.36.82.17:4329 , piece_id=KPGYZK5QIGRNMUHQLFRGRNIPI5EONZBHCGARRSDZCK2UFJLNN4TA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.664326608 (105h48m22.095406739s) [noise=true , address=104.36.82.11:4138 , piece_id=7L7M5DMAFNHOHVMUAQPFOKLU4TKIBL3TMS6D2F6X5T23BKDKGS7Q, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:29.202132828 (105h48m23.633212949s) [upload=110132]: Next returned nil segment Aug 13 03:14:29.202151514 (105h48m23.633231645s) [upload=110132]: Waiting for error group managing segments... Aug 13 03:14:29.624905241 (105h48m24.055985372s) [noise=true , address=77.90.31.14:4679 , piece_id=CFPI32OHVSZTT2YEOSL5ZXAHA37PT57VMTTTCTTIRQVMXAJM6D5A, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.625120442 (105h48m24.056200574s) [noise=true , address=104.36.82.59:4312 , piece_id=FZB6SMU32MTZTBAHUTHLWVZGQ53C67FAAIIH6VE2MTLS6PGFXXZQ, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.667080856 (105h48m24.098160987s) [noise=true , address=104.36.82.9:4024 , piece_id=Y5CY7VLCSC6BNGIH6NF6446BOU4LNIX5GCCY6DSWCPKXPCK4QDQQ, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.669032951 (105h48m24.100113082s) [noise=true , address=104.36.82.14:4250 , piece_id=CQY4HBMBDYVUKR4637GSB33VW6G4OEP53VH27DMSBPS54CATSBWA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.699132833 (105h48m24.130212964s) [noise=true , address=104.36.82.11:4138 , piece_id=7L7M5DMAFNHOHVMUAQPFOKLU4TKIBL3TMS6D2F6X5T23BKDKGS7Q, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.717982878 (105h48m24.149063009s) [noise=true , address=104.36.82.36:4637 , piece_id=QXP45IJF7253J7A5MY3C4DAA4VFFT5XULJ3Y4QTXTDCTVJP4IYGA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.724446195 (105h48m24.155526316s) [noise=true , address=104.36.82.37:4678 , piece_id=6JTI7SDS4DTDJFAF4FZCCCOON7KOPOXXT5MZ2MDIPSQXZZKF6TKQ, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.731783564 (105h48m24.162863695s) [noise=true , address=104.36.82.58:4284 , piece_id=K5J626LGT3TOFW6Q4G22RJJYQGXPYS2B74RB34HVPD6HTOQA2QCA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.780951378 (105h48m24.212031509s) [noise=true , address=104.36.82.46:4484 , piece_id=YRE57GHVKPCOHVRDGSL4FGGX4NPXUJF4L7ELHZGHJ4EC4QDMEU3A, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.794319577 (105h48m24.225399708s) [noise=true , address=104.36.82.47:4544 , piece_id=TIKP5LMX65MWSBUUPMX36JYRVFC3JVCA5KLRS7CIZMF3SJVS73OA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:30.252036350 (105h48m24.683116481s) [noise=true , address=79.127.219.38:4035 , piece_id=MXSRDYRUPY6DSE6N66WNQZPCC6KVBHISPOIZBGFAQSD3CUGSFWOA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:30.523524634 (105h48m24.954604765s) [noise=true , address=104.36.82.23:4472 , piece_id=IRJTNTCAIGVWU4HIE3L3ZCZH6VM6FHFU3SEZDXFYEV4R2BAP45RA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:31.207986103 (105h48m25.639066235s) [noise=true , address=77.90.31.10:4528 , piece_id=XKSFJ2I7HYD72TOTWFHDRRLTWE6SFSMKTDVKYIYFWI6VYQF4EW5Q, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:31.757453767 (105h48m26.188533898s) [noise=true , address=104.36.82.16:4299 , piece_id=CTNAIZNCPYLZLJZSH7OVSKBIJ2Q3J2QC75ONHCNPKVGRQEQWLL2Q, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:31.757477813 (105h48m26.188557934s) [seg_pos={0 0}, upload=110132]: Segment reached optimal threshold of 14 pieces. Aug 13 03:14:31.757561022 (105h48m26.188641153s) [noise=true , address=77.90.31.10:4530 , piece_id=JILOKHNOQCGIDXTL2NQF4HCNYE6RJXNH4OICDIVSHADVBIUVA4ZA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne): protocol: expected piece hash; context canceled; EOF Aug 13 03:14:31.757627769 (105h48m26.1887079s) [noise=true , address=104.36.82.18:4377 , piece_id=Y7NIH46TSTZBIMNAK6WLRFX6WYAZR2BPQLVKHJ4OK44TTD3JP2AA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JHy4o8dxFNeeiitLPavxspcf4rL9KfckcRAuC121v8YZG6qmT): protocol: expected piece hash; context canceled; EOF Aug 13 03:14:31.757682684 (105h48m26.188762815s) [noise=true , address=104.36.82.17:4329 , piece_id=KPGYZK5QIGRNMUHQLFRGRNIPI5EONZBHCGARRSDZCK2UFJLNN4TA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: expected piece hash; context canceled; EOF Aug 13 03:14:31.757703594 (105h48m26.188783725s) [noise=true , address=104.36.82.60:4352 , piece_id=UCDAKY7PVPM2LXMCEZYWZNQJV6DJRZIBB2VMJP2IB6N4ZEU6Q7IA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: write tcp 188.166.225.170:47910->104.36.82.60:4352: use of closed network connection; write tcp 188.166.225.170:47910->104.36.82.60:4352: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47910->104.36.82.60:4352: use of closed network connection Aug 13 03:14:31.757749713 (105h48m26.188829844s) [seg_pos={0 0}, upload=110132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 13 03:14:31.757833714 (105h48m26.188913845s) [upload=110132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 13 03:14:31.757837731 (105h48m26.188917862s) [upload=110132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:14:32.059045060 (105h48m26.490125191s) [upload=110132]: Done uploading segments... Aug 13 03:14:32.059061040 (105h48m26.490141171s) [upload=110132]: Upload finished. err: <nil> Duration: 5.80521807s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X..X.......X...........X...............X............] duration: 6.467777051s FAIL (2025-08-13 03:23:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 07a64daf5ce9db32d41aea6196fcc70ddc5670c6fb7b746f966c0a14f0f90cad != before e0a47ea874bf973a2cf983e7ff42ad385b8577ce8b4f3c6f68efa1aa67839818 ======LOGS====== Aug 13 03:14:27.200063120 (105h48m21.631143251s) [upload=110132]: Starting upload Aug 13 03:14:27.200107645 (105h48m21.631187776s) [upload=110132]: Done starting upload Aug 13 03:14:27.200237103 (105h48m21.631317234s) [upload=110132]: Uploading segments... Aug 13 03:14:27.200364997 (105h48m21.631445128s) [upload=110132]: Got next segment. Inline: false Aug 13 03:14:27.200415604 (105h48m21.631495785s) [upload=110132]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:14:27.200424100 (105h48m21.631504271s) [upload=110132]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 13 03:14:27.660811382 (105h48m22.091891513s) [seg_pos={0 0}, upload=110132]: Begin upload segment... Aug 13 03:14:27.660865877 (105h48m22.091946008s) [seg_pos={0 0}, upload=110132]: Done begin upload segment. Aug 13 03:14:27.660886587 (105h48m22.091966718s) [noise=true , address=77.90.31.10:4528 , piece_id=XKSFJ2I7HYD72TOTWFHDRRLTWE6SFSMKTDVKYIYFWI6VYQF4EW5Q, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.660900844 (105h48m22.091981055s) [noise=true , address=79.127.219.38:4035 , piece_id=MXSRDYRUPY6DSE6N66WNQZPCC6KVBHISPOIZBGFAQSD3CUGSFWOA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.661384269 (105h48m22.0924644s) [noise=true , address=104.36.82.58:4284 , piece_id=K5J626LGT3TOFW6Q4G22RJJYQGXPYS2B74RB34HVPD6HTOQA2QCA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.661598819 (105h48m22.09267894s) [noise=true , address=104.36.82.14:4250 , piece_id=CQY4HBMBDYVUKR4637GSB33VW6G4OEP53VH27DMSBPS54CATSBWA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.661669895 (105h48m22.092750026s) [noise=true , address=104.36.82.47:4544 , piece_id=TIKP5LMX65MWSBUUPMX36JYRVFC3JVCA5KLRS7CIZMF3SJVS73OA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.661996209 (105h48m22.09307634s) [noise=true , address=77.90.31.14:4679 , piece_id=CFPI32OHVSZTT2YEOSL5ZXAHA37PT57VMTTTCTTIRQVMXAJM6D5A, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.662169181 (105h48m22.093249342s) [noise=true , address=77.90.31.10:4530 , piece_id=JILOKHNOQCGIDXTL2NQF4HCNYE6RJXNH4OICDIVSHADVBIUVA4ZA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.662275895 (105h48m22.093356016s) [noise=true , address=104.36.82.18:4377 , piece_id=Y7NIH46TSTZBIMNAK6WLRFX6WYAZR2BPQLVKHJ4OK44TTD3JP2AA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.662522045 (105h48m22.093602166s) [noise=true , address=104.36.82.9:4024 , piece_id=Y5CY7VLCSC6BNGIH6NF6446BOU4LNIX5GCCY6DSWCPKXPCK4QDQQ, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.662779709 (105h48m22.09385984s) [noise=true , address=104.36.82.46:4484 , piece_id=YRE57GHVKPCOHVRDGSL4FGGX4NPXUJF4L7ELHZGHJ4EC4QDMEU3A, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.662985592 (105h48m22.094065723s) [noise=true , address=104.36.82.59:4312 , piece_id=FZB6SMU32MTZTBAHUTHLWVZGQ53C67FAAIIH6VE2MTLS6PGFXXZQ, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.663176938 (105h48m22.094257069s) [noise=true , address=104.36.82.60:4352 , piece_id=UCDAKY7PVPM2LXMCEZYWZNQJV6DJRZIBB2VMJP2IB6N4ZEU6Q7IA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.663367453 (105h48m22.094447584s) [noise=true , address=104.36.82.16:4299 , piece_id=CTNAIZNCPYLZLJZSH7OVSKBIJ2Q3J2QC75ONHCNPKVGRQEQWLL2Q, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.663568779 (105h48m22.09464891s) [noise=true , address=104.36.82.37:4678 , piece_id=6JTI7SDS4DTDJFAF4FZCCCOON7KOPOXXT5MZ2MDIPSQXZZKF6TKQ, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.663767839 (105h48m22.09484797s) [noise=true , address=104.36.82.36:4637 , piece_id=QXP45IJF7253J7A5MY3C4DAA4VFFT5XULJ3Y4QTXTDCTVJP4IYGA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.663960167 (105h48m22.095040288s) [noise=true , address=104.36.82.23:4472 , piece_id=IRJTNTCAIGVWU4HIE3L3ZCZH6VM6FHFU3SEZDXFYEV4R2BAP45RA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.664144580 (105h48m22.095224711s) [noise=true , address=104.36.82.17:4329 , piece_id=KPGYZK5QIGRNMUHQLFRGRNIPI5EONZBHCGARRSDZCK2UFJLNN4TA, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:27.664326608 (105h48m22.095406739s) [noise=true , address=104.36.82.11:4138 , piece_id=7L7M5DMAFNHOHVMUAQPFOKLU4TKIBL3TMS6D2F6X5T23BKDKGS7Q, seg_pos={0 0}, upload=110132]: Uploading piece... Aug 13 03:14:29.202132828 (105h48m23.633212949s) [upload=110132]: Next returned nil segment Aug 13 03:14:29.202151514 (105h48m23.633231645s) [upload=110132]: Waiting for error group managing segments... Aug 13 03:14:29.624905241 (105h48m24.055985372s) [noise=true , address=77.90.31.14:4679 , piece_id=CFPI32OHVSZTT2YEOSL5ZXAHA37PT57VMTTTCTTIRQVMXAJM6D5A, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.625120442 (105h48m24.056200574s) [noise=true , address=104.36.82.59:4312 , piece_id=FZB6SMU32MTZTBAHUTHLWVZGQ53C67FAAIIH6VE2MTLS6PGFXXZQ, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.667080856 (105h48m24.098160987s) [noise=true , address=104.36.82.9:4024 , piece_id=Y5CY7VLCSC6BNGIH6NF6446BOU4LNIX5GCCY6DSWCPKXPCK4QDQQ, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.669032951 (105h48m24.100113082s) [noise=true , address=104.36.82.14:4250 , piece_id=CQY4HBMBDYVUKR4637GSB33VW6G4OEP53VH27DMSBPS54CATSBWA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.699132833 (105h48m24.130212964s) [noise=true , address=104.36.82.11:4138 , piece_id=7L7M5DMAFNHOHVMUAQPFOKLU4TKIBL3TMS6D2F6X5T23BKDKGS7Q, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.717982878 (105h48m24.149063009s) [noise=true , address=104.36.82.36:4637 , piece_id=QXP45IJF7253J7A5MY3C4DAA4VFFT5XULJ3Y4QTXTDCTVJP4IYGA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.724446195 (105h48m24.155526316s) [noise=true , address=104.36.82.37:4678 , piece_id=6JTI7SDS4DTDJFAF4FZCCCOON7KOPOXXT5MZ2MDIPSQXZZKF6TKQ, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.731783564 (105h48m24.162863695s) [noise=true , address=104.36.82.58:4284 , piece_id=K5J626LGT3TOFW6Q4G22RJJYQGXPYS2B74RB34HVPD6HTOQA2QCA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.780951378 (105h48m24.212031509s) [noise=true , address=104.36.82.46:4484 , piece_id=YRE57GHVKPCOHVRDGSL4FGGX4NPXUJF4L7ELHZGHJ4EC4QDMEU3A, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:29.794319577 (105h48m24.225399708s) [noise=true , address=104.36.82.47:4544 , piece_id=TIKP5LMX65MWSBUUPMX36JYRVFC3JVCA5KLRS7CIZMF3SJVS73OA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:30.252036350 (105h48m24.683116481s) [noise=true , address=79.127.219.38:4035 , piece_id=MXSRDYRUPY6DSE6N66WNQZPCC6KVBHISPOIZBGFAQSD3CUGSFWOA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:30.523524634 (105h48m24.954604765s) [noise=true , address=104.36.82.23:4472 , piece_id=IRJTNTCAIGVWU4HIE3L3ZCZH6VM6FHFU3SEZDXFYEV4R2BAP45RA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:31.207986103 (105h48m25.639066235s) [noise=true , address=77.90.31.10:4528 , piece_id=XKSFJ2I7HYD72TOTWFHDRRLTWE6SFSMKTDVKYIYFWI6VYQF4EW5Q, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:31.757453767 (105h48m26.188533898s) [noise=true , address=104.36.82.16:4299 , piece_id=CTNAIZNCPYLZLJZSH7OVSKBIJ2Q3J2QC75ONHCNPKVGRQEQWLL2Q, seg_pos={0 0}, upload=110132]: Done uploading piece. err: <nil> Aug 13 03:14:31.757477813 (105h48m26.188557934s) [seg_pos={0 0}, upload=110132]: Segment reached optimal threshold of 14 pieces. Aug 13 03:14:31.757561022 (105h48m26.188641153s) [noise=true , address=77.90.31.10:4530 , piece_id=JILOKHNOQCGIDXTL2NQF4HCNYE6RJXNH4OICDIVSHADVBIUVA4ZA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne): protocol: expected piece hash; context canceled; EOF Aug 13 03:14:31.757627769 (105h48m26.1887079s) [noise=true , address=104.36.82.18:4377 , piece_id=Y7NIH46TSTZBIMNAK6WLRFX6WYAZR2BPQLVKHJ4OK44TTD3JP2AA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JHy4o8dxFNeeiitLPavxspcf4rL9KfckcRAuC121v8YZG6qmT): protocol: expected piece hash; context canceled; EOF Aug 13 03:14:31.757682684 (105h48m26.188762815s) [noise=true , address=104.36.82.17:4329 , piece_id=KPGYZK5QIGRNMUHQLFRGRNIPI5EONZBHCGARRSDZCK2UFJLNN4TA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ht9BFK3Y12gKG8i1Szefj4jkP71hgXZ5XLT4schym7GX5ycW): protocol: expected piece hash; context canceled; EOF Aug 13 03:14:31.757703594 (105h48m26.188783725s) [noise=true , address=104.36.82.60:4352 , piece_id=UCDAKY7PVPM2LXMCEZYWZNQJV6DJRZIBB2VMJP2IB6N4ZEU6Q7IA, seg_pos={0 0}, upload=110132]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RtYD3H9egoiNUbXsxJMWetAHYqjKGtfnGyNLSn6tbugdHaLXi): protocol: write tcp 188.166.225.170:47910->104.36.82.60:4352: use of closed network connection; write tcp 188.166.225.170:47910->104.36.82.60:4352: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47910->104.36.82.60:4352: use of closed network connection Aug 13 03:14:31.757749713 (105h48m26.188829844s) [seg_pos={0 0}, upload=110132]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 13 03:14:31.757833714 (105h48m26.188913845s) [upload=110132]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 13 03:14:31.757837731 (105h48m26.188917862s) [upload=110132]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:14:32.059045060 (105h48m26.490125191s) [upload=110132]: Done uploading segments... Aug 13 03:14:32.059061040 (105h48m26.490141171s) [upload=110132]: Upload finished. err: <nil> Duration: 6.467805595s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 1.863458887s FAIL (2025-08-13 03:22:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Aug 13 03:06:41.898033585 (105h40m36.329113716s) [upload=110007]: Starting upload Aug 13 03:06:41.898093971 (105h40m36.329174102s) [upload=110007]: Done starting upload Aug 13 03:06:41.898140650 (105h40m36.329220781s) [upload=110007]: Uploading segments... Aug 13 03:06:41.898172331 (105h40m36.329252462s) [upload=110007]: Got next segment. Inline: true Aug 13 03:06:41.898176078 (105h40m36.329256209s) [upload=110007]: Waiting for error group managing segments... Aug 13 03:06:41.898208831 (105h40m36.329288962s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 13 03:06:41.898212778 (105h40m36.329292909s) [upload=110007]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 13 03:06:41.898215563 (105h40m36.329295694s) [upload=110007]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 13 03:07:11.244546166 (105h41m5.675626307s) [upload=110007]: Done uploading segments... Aug 13 03:07:11.244585330 (105h41m5.675665482s) [upload=110007]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.863497031s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)