Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 23 out of 60 is failed (threshold=3) [......................XX.........XX.XX.XXXX.XX.XXXXX.X.XXXXX] duration: 30.233738525s FAIL (2025-07-26 22:03:48 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.233754736s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 34 out of 60 is failed (threshold=3) [..........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.23285521s FAIL (2025-07-26 22:03:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 4) ======LOGS====== , seg_pos={0 0}, upload=56094]: Uploading piece... Jul 26 22:02:48.282233472 (58h26m58.004796512s) [noise=true , address=79.127.219.38:4015 , piece_id=URPSO5GUCTWLB7CXCJT3XX7WZPESG542WFVTT7BBHTPR3XXEM3GQ, seg_pos={0 0}, upload=56094]: Uploading piece... Jul 26 22:02:48.282393618 (58h26m58.004956658s) [noise=true , address=77.90.31.21:4905 , piece_id=2BKJQAO4ICK3VF3XY2DEP7CWPKBN5O5O6VISL37HZCZDZ3PTKSIA, seg_pos={0 0}, upload=56094]: Uploading piece... Jul 26 22:02:48.282448061 (58h26m58.005011101s) [noise=true , address=104.36.82.10:4084 , piece_id=MCFZSNAEM6SZROWHPL67G5GOQN2MG24VMFK3MMRYGARCGQAK4N7A, seg_pos={0 0}, upload=56094]: Uploading piece... Jul 26 22:02:48.282653445 (58h26m58.005216485s) [noise=true , address=104.36.82.9:4038 , piece_id=TLQRY37I3TKW2BPSRXFUMTWKI53SH4TNQTPHDAX5AVV47M4GWIBA, seg_pos={0 0}, upload=56094]: Uploading piece... Jul 26 22:02:48.282760430 (58h26m58.00532347s) [noise=true , address=104.36.82.56:4235 , piece_id=CHRKBAREZPBHZREDRNCNWAE4LTGD6SIXOAGSBLWW3CU3A7OKGY3Q, seg_pos={0 0}, upload=56094]: Uploading piece... Jul 26 22:02:48.282844410 (58h26m58.00540744s) [noise=true , address=104.36.82.39:4746 , piece_id=3GALYJYJCO5WZKO26R2HP75GH25YNB4IQNULABSWR56R4WJYLFWA, seg_pos={0 0}, upload=56094]: Uploading piece... Jul 26 22:02:48.283084159 (58h26m58.005647199s) [noise=true , address=104.36.82.53:4142 , piece_id=OZZ46C5N3563FY766YDPOUUKSYBFLXBQGEL3EF65S2R3NMH6FKJQ, seg_pos={0 0}, upload=56094]: Uploading piece... Jul 26 22:02:48.283154543 (58h26m58.005717593s) [noise=true , address=45.91.116.13:4187 , piece_id=6NCUN6LKMRP2N7L2D4Q4MTJB52PNSEQWKDDC625BN3GO57DN2EQA, seg_pos={0 0}, upload=56094]: Uploading piece... Jul 26 22:02:48.283559609 (58h26m58.006122649s) [noise=true , address=104.36.82.6:4878 , piece_id=3DZUNVVEJONPJ6M7QZFT2ZXRY2VFSVBZSYBOAXTFYCCTEOVX23EA, seg_pos={0 0}, upload=56094]: Uploading piece... Jul 26 22:02:49.773858572 (58h26m59.496421602s) [upload=56094]: Next returned nil segment Jul 26 22:02:49.773881315 (58h26m59.496444355s) [upload=56094]: Waiting for error group managing segments... Jul 26 22:02:50.353152530 (58h27m0.07571557s) [noise=true , address=79.127.219.38:4015 , piece_id=URPSO5GUCTWLB7CXCJT3XX7WZPESG542WFVTT7BBHTPR3XXEM3GQ, seg_pos={0 0}, upload=56094]: Done uploading piece. err: <nil> Jul 26 22:02:53.034483690 (58h27m2.75704673s) [noise=true , address=45.91.116.13:4187 , piece_id=6NCUN6LKMRP2N7L2D4Q4MTJB52PNSEQWKDDC625BN3GO57DN2EQA, seg_pos={0 0}, upload=56094]: Done uploading piece. err: <nil> Jul 26 22:02:58.657858226 (58h27m8.380421267s) [noise=true , address=77.90.31.21:4905 , piece_id=2BKJQAO4ICK3VF3XY2DEP7CWPKBN5O5O6VISL37HZCZDZ3PTKSIA, seg_pos={0 0}, upload=56094]: Done uploading piece. err: <nil> Jul 26 22:02:58.761636336 (58h27m8.484199366s) [noise=true , address=79.127.201.215:4344 , piece_id=6IG3JUHXJHGITI5354BPBZ3NGJPCO4Y2CDIZQRONKWLUQTNQFJFQ, seg_pos={0 0}, upload=56094]: Done uploading piece. err: <nil> Jul 26 22:03:47.194070794 (58h27m56.916633834s) [noise=true , address=104.36.82.43:4864 , piece_id=LNZ3Z6PPSWOVWEIGUPNQJJ6QAVTBSRLJM4MLIHBBGJB73S7Z5KPQ, seg_pos={0 0}, upload=56094]: Done uploading piece. err: ecclient: upload failed (node:1z6uPcGsR78PiEpT46NBJEdyH33Re1YU5CtrMMFVeFf1KD3egC, address:104.36.82.43:4864): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:03:47.194333417 (58h27m56.916896457s) [noise=true , address=104.36.82.56:4235 , piece_id=CHRKBAREZPBHZREDRNCNWAE4LTGD6SIXOAGSBLWW3CU3A7OKGY3Q, seg_pos={0 0}, upload=56094]: Done uploading piece. err: ecclient: upload failed (node:1a1sW1G1WPL5WQ2XxprWKQMsJkcb5ETSkk3ULp1TjRaZWptaT, address:104.36.82.56:4235): protocol: write tcp 188.166.225.170:48008->104.36.82.56:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:03:47.194368324 (58h27m56.916931364s) [noise=true , address=104.36.82.55:4219 , piece_id=STONDNXPE7SPPWLR65S3EBCTL2KZU33UJ62VXSS4FHDQO4OCI5KQ, seg_pos={0 0}, upload=56094]: Done uploading piece. err: ecclient: upload failed (node:12TqSxiG3LYYc3W5T6wqiuuKwoh1LvCdntou7ve6TAEbXYf2QKn, address:104.36.82.55:4219): protocol: write tcp 188.166.225.170:50280->104.36.82.55:4219: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:03:47.194395626 (58h27m56.916958666s) [noise=true , address=104.36.82.36:4647 , piece_id=SOKKW5ZPATEHYI5AKXYUYMRP2OCTXVCPJ3GNS5NSWCBGJRT5Y6QQ, seg_pos={0 0}, upload=56094]: Done uploading piece. err: ecclient: upload failed (node:1arYSsvaPDeLVdppvVEB5VYwRAcJkLfYWXMG7CAHvvY4YTKxpZ, address:104.36.82.36:4647): protocol: write tcp 188.166.225.170:41974->104.36.82.36:4647: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:03:47.194426855 (58h27m56.916989895s) [noise=true , address=104.36.82.46:4493 , piece_id=AY7AG4Q7R3JPQE7V2KYWCQXA72VREIFE6ZG2RLKERKVNNELMSRLQ, seg_pos={0 0}, upload=56094]: Done uploading piece. err: ecclient: upload failed (node:127drPYZWm7RRXuCLtuzDgSaxxuu3k1tg6eo3KWtrQGjBaUbfRJ, address:104.36.82.46:4493): protocol: write tcp 188.166.225.170:51164->104.36.82.46:4493: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:03:47.194455941 (58h27m56.917018981s) [noise=true , address=104.36.82.10:4091 , piece_id=MPRETUNPTFGUUSTR6DLMSYA4WTKIJNVK4J3QM5SYUHZSZ6BOM7JA, seg_pos={0 0}, upload=56094]: Done uploading piece. err: ecclient: upload failed (node:1yyUzkigTS9m3bDrZtAFL7tciNxpiCNswBSfBqXn1UW83GMLwi, address:104.36.82.10:4091): protocol: write tcp 188.166.225.170:33932->104.36.82.10:4091: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:03:47.194483925 (58h27m56.917046965s) [noise=true , address=104.36.82.9:4038 , piece_id=TLQRY37I3TKW2BPSRXFUMTWKI53SH4TNQTPHDAX5AVV47M4GWIBA, seg_pos={0 0}, upload=56094]: Done uploading piece. err: ecclient: upload failed (node:12VTVM9YuevPA4vsWExjpyW5B9pqyEKKruUwnj1jNhE3j1ScCBg, address:104.36.82.9:4038): protocol: write tcp 188.166.225.170:48284->104.36.82.9:4038: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:03:47.194495517 (58h27m56.917058557s) [noise=true , address=104.36.82.52:4111 , piece_id=4LPXFXDSYV6XMTHC3LNSLXXE323V2ZRT45ZJYRYMMALHL6R2Y6GA, seg_pos={0 0}, upload=56094]: Done uploading piece. err: ecclient: upload failed (node:12krqZcFoBPfLFbLGQUAjPMzHJ4HKqsKPRg6Xap7nuShaEKBumH, address:104.36.82.52:4111): protocol: write tcp 188.166.225.170:38890->104.36.82.52:4111: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:03:47.194516748 (58h27m56.917079778s) [noise=true , address=104.36.82.39:4746 , piece_id=3GALYJYJCO5WZKO26R2HP75GH25YNB4IQNULABSWR56R4WJYLFWA, seg_pos={0 0}, upload=56094]: Done uploading piece. err: ecclient: upload failed (node:12SBP5cx63KB8oMuzhuq78a1V88januH2X5K3MzJGdXxFvXD4VM, address:104.36.82.39:4746): protocol: write tcp 188.166.225.170:56828->104.36.82.39:4746: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:03:47.194533429 (58h27m56.917096469s) [noise=true , address=104.36.82.50:4034 , piece_id=XEZI3VM2A3B6EPWJL7WWUEPAXMF3LKVZ7VRNUGTOKOJVGXEOJJEQ, seg_pos={0 0}, upload=56094]: Done uploading piece. err: ecclient: upload failed (node:12q2ocJMc2bzMsrFH4heg52ZBziCHNxQc7cxDGU79AeUZxqoB9r, address:104.36.82.50:4034): protocol: write tcp 188.166.225.170:34124->104.36.82.50:4034: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:03:47.194546695 (58h27m56.917109725s) [noise=true , address=104.36.82.53:4142 , piece_id=OZZ46C5N3563FY766YDPOUUKSYBFLXBQGEL3EF65S2R3NMH6FKJQ, seg_pos={0 0}, upload=56094]: Done uploading piece. err: ecclient: upload failed (node:1yLPWJs1RfZgFisHUpW3wwiFsvV98yz7adKtAKAK3unq1KZCSK, address:104.36.82.53:4142): protocol: write tcp 188.166.225.170:45048->104.36.82.53:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:03:47.194564208 (58h27m56.917127248s) [noise=true , address=104.36.82.10:4084 , piece_id=MCFZSNAEM6SZROWHPL67G5GOQN2MG24VMFK3MMRYGARCGQAK4N7A, seg_pos={0 0}, upload=56094]: Done uploading piece. err: ecclient: upload failed (node:1mELnhMiRJ5GkUWYEggdxhCQ2JnDbT7qEMSiAuMQsF7BTnwP1u, address:104.36.82.10:4084): protocol: write tcp 188.166.225.170:45254->104.36.82.10:4084: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:03:47.194574719 (58h27m56.917137759s) [noise=true , address=104.36.82.47:4531 , piece_id=FHIIRXX3UGETTFGZWGGZHHDN75PCSNPX3O4I3EYRZXCXEOHL7GTA, seg_pos={0 0}, upload=56094]: Done uploading piece. err: ecclient: upload failed (node:12LZprfyqfD2YUwgAarzyjbgHEb6jMhAMAday5C45xk3j9RUS4u, address:104.36.82.47:4531): protocol: write tcp 188.166.225.170:45028->104.36.82.47:4531: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:03:47.194594957 (58h27m56.917157997s) [noise=true , address=104.36.82.6:4878 , piece_id=3DZUNVVEJONPJ6M7QZFT2ZXRY2VFSVBZSYBOAXTFYCCTEOVX23EA, seg_pos={0 0}, upload=56094]: Done uploading piece. err: ecclient: upload failed (node:126ZeAKnpjpxa3MgCBU3KZKBDdn8MfZgiixE4zR5fZpPwaQj9NC, address:104.36.82.6:4878): protocol: write tcp 188.166.225.170:58388->104.36.82.6:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:03:47.194615847 (58h27m56.917178887s) [seg_pos={0 0}, upload=56094]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil> Jul 26 22:03:47.423893036 (58h27m57.146456076s) [upload=56094]: Done uploading segments... Jul 26 22:03:47.423920719 (58h27m57.146483759s) [upload=56094]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4) Duration: 1m0.232866913s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [...............X....X......XX....XX.XX.XXX.X.XX.XXX.XXXXXX..] duration: 10.742203792s FAIL (2025-07-26 22:03:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== CEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:42362->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:42362->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42362->77.90.31.10:4517: use of closed network connection Jul 26 22:01:56.285677843 (58h26m6.008240883s) [seg_pos={0 0}, upload=56076]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:01:56.513982741 (58h26m6.236545781s) [upload=56076]: Done uploading segments... Jul 26 22:01:56.514008370 (58h26m6.23657141s) [upload=56076]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 26 22:01:57.390344722 (58h26m7.112907762s) [upload=56082]: Starting upload Jul 26 22:01:57.390413313 (58h26m7.112976353s) [upload=56082]: Done starting upload Jul 26 22:01:57.390523063 (58h26m7.113086103s) [upload=56082]: Uploading segments... Jul 26 22:01:57.390607184 (58h26m7.113170224s) [upload=56082]: Got next segment. Inline: false Jul 26 22:01:57.390634677 (58h26m7.113197717s) [upload=56082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:01:57.390650066 (58h26m7.113213106s) [upload=56082]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:01:57.832734230 (58h26m7.55529727s) [seg_pos={0 0}, upload=56082]: Begin upload segment... Jul 26 22:01:57.832776451 (58h26m7.555339491s) [seg_pos={0 0}, upload=56082]: Done begin upload segment. Jul 26 22:01:57.832800907 (58h26m7.555363947s) [noise=true , address=45.91.116.13:4154 , piece_id=AISKJI4VJHWUGCUERMPZYAV2PVN5CLMK5FSW7MV3PAODNRDIZEUA, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:57.832834942 (58h26m7.555397982s) [noise=true , address=45.91.116.8:4045 , piece_id=TUZS6EVEZ62TJJ7KD2WJ45AJKYDC3U4X642ZXANXRXWXWNLBM3XA, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:57.833157620 (58h26m7.55572066s) [noise=true , address=45.91.116.12:4227 , piece_id=FFEJZEVNZHWCQ5XY7MAPVJDZJF4SBGTQNJZI6CTAQBYOJJWPQ7FA, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:57.833205873 (58h26m7.555768913s) [noise=true , address=45.91.116.9:4262 , piece_id=P64ZMXR2I5VBJQLD6IN3OBNMTZRXU22LJ5FZXCVD7XQFE62RPS6Q, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:57.833414842 (58h26m7.555977882s) [noise=true , address=77.90.31.10:4519 , piece_id=IGYIDV75DZSUT5QBCPYSL7OTDBOUN2CPAGPKA7LUI6OGG34ATEPQ, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:57.833504374 (58h26m7.556067404s) [noise=true , address=45.91.116.5:4298 , piece_id=BSLK2XR7HC236XWVMTZ67MDEEOD6C3XIXNFBXRQVUMCU234NYNKA, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:57.833656244 (58h26m7.556219284s) [noise=true , address=45.91.116.4:4938 , piece_id=7BXISFB73QQVKKSLQNDWR2OJI3CQJ3ZK3XMM3QCWQ3CDK4ZLY5IA, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:57.833826269 (58h26m7.556389309s) [noise=true , address=104.36.82.7:4909 , piece_id=E7N4ASRRZVVPF27Z2QTGESCI76SRYQHLL44LQLYH2GEFZGLYZAJQ, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:57.833922884 (58h26m7.556485924s) [noise=true , address=77.90.31.16:4735 , piece_id=YB5ZNU7M7D5FQZWIRO67B3VBTFDNQB4UIKTVFAIZ2J3GKRUOV7NQ, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:57.834167713 (58h26m7.556730753s) [noise=true , address=104.36.82.5:4837 , piece_id=53PAGMESJHS3IYYTAVGCYDBOHDRQLQPVYB3EB423DNJMI5FMZATQ, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:57.834230172 (58h26m7.556793212s) [noise=true , address=45.91.116.7:4009 , piece_id=6BSRDBUMTNALWY3KHN4HXIL3DJHC6PXAQASZ6DPS5CG6D5CPRMEA, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:57.834367896 (58h26m7.556930936s) [noise=true , address=104.36.82.16:4293 , piece_id=QZGF3ASFDALJCFCT7OYDJBN5GYCKNGVKHKQE277ZU62RW3Q2TZSA, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:57.834472486 (58h26m7.557035526s) [noise=true , address=104.36.82.15:4258 , piece_id=CMRIEATD2Q3Y7EDSH4ITUVCI7XWRS4RKTUGID4BJHBROJR3PGXMQ, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:57.834555004 (58h26m7.557118044s) [noise=true , address=77.90.31.12:4592 , piece_id=5CZXLU63WV2OJMMJBC3ALOKKKT7XGTDIUJGLL3B76HBFWV3DOASA, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:57.834703618 (58h26m7.557266648s) [noise=true , address=45.91.116.6:4974 , piece_id=NGESKWVBJ5O736MPOY5R53FHVMSRCXVEZQRCXUSZCOKBHNZJYOMQ, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:57.834745158 (58h26m7.557308198s) [noise=true , address=45.91.116.11:4117 , piece_id=PBEWSLZWLB6DYR2R4WYGOJPCLSA4KOUHVDLYKBGGHQBQGYELZ6ZQ, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:59.573575768 (58h26m9.296138808s) [upload=56082]: Next returned nil segment Jul 26 22:01:59.573593682 (58h26m9.296156712s) [upload=56082]: Waiting for error group managing segments... Jul 26 22:01:59.849382632 (58h26m9.571945672s) [noise=true , address=45.91.116.13:4154 , piece_id=AISKJI4VJHWUGCUERMPZYAV2PVN5CLMK5FSW7MV3PAODNRDIZEUA, seg_pos={0 0}, upload=56082]: Done uploading piece. err: <nil> Jul 26 22:01:59.859793060 (58h26m9.58235609s) [noise=true , address=45.91.116.8:4045 , piece_id=TUZS6EVEZ62TJJ7KD2WJ45AJKYDC3U4X642ZXANXRXWXWNLBM3XA, seg_pos={0 0}, upload=56082]: Done uploading piece. err: <nil> Jul 26 22:01:59.923504861 (58h26m9.646067901s) [noise=true , address=45.91.116.5:4298 , piece_id=BSLK2XR7HC236XWVMTZ67MDEEOD6C3XIXNFBXRQVUMCU234NYNKA, seg_pos={0 0}, upload=56082]: Done uploading piece. err: <nil> Jul 26 22:01:59.937899672 (58h26m9.660462712s) [noise=true , address=45.91.116.11:4117 , piece_id=PBEWSLZWLB6DYR2R4WYGOJPCLSA4KOUHVDLYKBGGHQBQGYELZ6ZQ, seg_pos={0 0}, upload=56082]: Done uploading piece. err: <nil> Jul 26 22:01:59.953884096 (58h26m9.676447136s) [noise=true , address=45.91.116.7:4009 , piece_id=6BSRDBUMTNALWY3KHN4HXIL3DJHC6PXAQASZ6DPS5CG6D5CPRMEA, seg_pos={0 0}, upload=56082]: Done uploading piece. err: <nil> Jul 26 22:02:00.027736248 (58h26m9.750299288s) [noise=true , address=45.91.116.4:4938 , piece_id=7BXISFB73QQVKKSLQNDWR2OJI3CQJ3ZK3XMM3QCWQ3CDK4ZLY5IA, seg_pos={0 0}, upload=56082]: Done uploading piece. err: <nil> Jul 26 22:02:00.034667830 (58h26m9.75723087s) [noise=true , address=45.91.116.9:4262 , piece_id=P64ZMXR2I5VBJQLD6IN3OBNMTZRXU22LJ5FZXCVD7XQFE62RPS6Q, seg_pos={0 0}, upload=56082]: Done uploading piece. err: <nil> Jul 26 22:02:00.082456705 (58h26m9.805019745s) [noise=true , address=45.91.116.12:4227 , piece_id=FFEJZEVNZHWCQ5XY7MAPVJDZJF4SBGTQNJZI6CTAQBYOJJWPQ7FA, seg_pos={0 0}, upload=56082]: Done uploading piece. err: <nil> Jul 26 22:02:00.089126276 (58h26m9.811689316s) [noise=true , address=45.91.116.6:4974 , piece_id=NGESKWVBJ5O736MPOY5R53FHVMSRCXVEZQRCXUSZCOKBHNZJYOMQ, seg_pos={0 0}, upload=56082]: Done uploading piece. err: <nil> Jul 26 22:02:05.325925773 (58h26m15.048488813s) [noise=true , address=77.90.31.10:4519 , piece_id=IGYIDV75DZSUT5QBCPYSL7OTDBOUN2CPAGPKA7LUI6OGG34ATEPQ, seg_pos={0 0}, upload=56082]: Done uploading piece. err: <nil> Jul 26 22:02:22.010826242 (58h26m31.733389282s) [noise=true , address=77.90.31.12:4592 , piece_id=5CZXLU63WV2OJMMJBC3ALOKKKT7XGTDIUJGLL3B76HBFWV3DOASA, seg_pos={0 0}, upload=56082]: Done uploading piece. err: <nil> Jul 26 22:02:26.743265063 (58h26m36.465828103s) [noise=true , address=104.36.82.15:4258 , piece_id=CMRIEATD2Q3Y7EDSH4ITUVCI7XWRS4RKTUGID4BJHBROJR3PGXMQ, seg_pos={0 0}, upload=56082]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:54148->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:02:26.743406403 (58h26m36.465969443s) [noise=true , address=104.36.82.16:4293 , piece_id=QZGF3ASFDALJCFCT7OYDJBN5GYCKNGVKHKQE277ZU62RW3Q2TZSA, seg_pos={0 0}, upload=56082]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:02:26.743464755 (58h26m36.466027795s) [noise=true , address=77.90.31.16:4735 , piece_id=YB5ZNU7M7D5FQZWIRO67B3VBTFDNQB4UIKTVFAIZ2J3GKRUOV7NQ, seg_pos={0 0}, upload=56082]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:02:26.743648857 (58h26m36.466211897s) [noise=true , address=104.36.82.7:4909 , piece_id=E7N4ASRRZVVPF27Z2QTGESCI76SRYQHLL44LQLYH2GEFZGLYZAJQ, seg_pos={0 0}, upload=56082]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 188.166.225.170:37230->104.36.82.7:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:02:26.743655089 (58h26m36.466218129s) [noise=true , address=104.36.82.5:4837 , piece_id=53PAGMESJHS3IYYTAVGCYDBOHDRQLQPVYB3EB423DNJMI5FMZATQ, seg_pos={0 0}, upload=56082]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 188.166.225.170:43472->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:02:26.743682802 (58h26m36.466245832s) [seg_pos={0 0}, upload=56082]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 22:02:26.983797456 (58h26m36.706360496s) [upload=56082]: Done uploading segments... Jul 26 22:02:26.983823416 (58h26m36.706386456s) [upload=56082]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 10.742227657s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 1.852465721s FAIL (2025-07-26 22:03:32 UTC) orbiter warning: digest mismatch: after 0783c6a051cf3b535b7c61f8e6f0cfe751970c7121d9bce2182fd51e29ca8fc6 != before ee59801cb45aa958945ed73fcee20d9c3604e7f558c75f49df119aca36a29d88 Duration: 1.8524858s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 22 out of 60 is failed (threshold=3) [.......................XX.........XX.XX.XXXX.XX.XXXXX.X.XXXX] duration: 30.283066149s FAIL (2025-07-26 22:03:18 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.283086628s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 1.828483294s FAIL (2025-07-26 22:02:51 UTC) orbiter warning: digest mismatch: after 0783c6a051cf3b535b7c61f8e6f0cfe751970c7121d9bce2182fd51e29ca8fc6 != before ee59801cb45aa958945ed73fcee20d9c3604e7f558c75f49df119aca36a29d88 Duration: 1.828509014s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 33 out of 60 is failed (threshold=3) [...........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.231944115s FAIL (2025-07-26 22:02:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 5) ======LOGS====== LBYR3N6L2CIBAXZNDXDB7QS6Q, seg_pos={0 0}, upload=56080]: Uploading piece... Jul 26 22:01:47.830802436 (58h25m57.553365476s) [noise=true , address=104.36.82.18:4390 , piece_id=P5CYDJ5KKWCAOYQFYDTFJHRNJ7B2VV56CG6SFR5N3NFYPTM62MJQ, seg_pos={0 0}, upload=56080]: Uploading piece... Jul 26 22:01:47.830920602 (58h25m57.553483652s) [noise=true , address=104.36.82.56:4231 , piece_id=ZQD3ZSQP2EFLRN2COOZGQ4NFCLDZ74UKGKNILGB2AUKZTL3H3LVQ, seg_pos={0 0}, upload=56080]: Uploading piece... Jul 26 22:01:47.831094325 (58h25m57.553657365s) [noise=true , address=104.36.82.46:4482 , piece_id=3FN3UKJGGNEZJRGAKHFAFHAKRUARCSIGDFEMKGOKEO4D2KCR2HGA, seg_pos={0 0}, upload=56080]: Uploading piece... Jul 26 22:01:47.831320728 (58h25m57.553883768s) [noise=true , address=104.36.82.55:4200 , piece_id=OT7A57QX7R72N3OLAN2SMY6OKOXFM3S3G6YHQV5WVZUEQNRC2JPQ, seg_pos={0 0}, upload=56080]: Uploading piece... Jul 26 22:01:47.831334474 (58h25m57.553897514s) [noise=true , address=104.36.82.53:4145 , piece_id=FHZISQBV767WJUXATGQLZV3EWY3XP64VBWHINE6N63AELXSAD5HQ, seg_pos={0 0}, upload=56080]: Uploading piece... Jul 26 22:01:47.831533395 (58h25m57.554096435s) [noise=true , address=79.127.205.234:4605 , piece_id=SP4QTS45Z5EJ7OOISC2QD4LGRQNBBZAKNMPQZNXFL5OACQ6UWKZQ, seg_pos={0 0}, upload=56080]: Uploading piece... Jul 26 22:01:47.831669776 (58h25m57.554232816s) [noise=true , address=104.36.82.52:4100 , piece_id=WFL4OIJZTEHACRRD46IRJOXOYTHIULU7OBBJRAZB5R6ARDSR4LTA, seg_pos={0 0}, upload=56080]: Uploading piece... Jul 26 22:01:47.831747314 (58h25m57.554310434s) [noise=true , address=104.36.82.49:4979 , piece_id=FZAUGYJVNLXE4ENXIATPI6W4FCTOT7NXWFFOCSWZ2OYROLZCR2AQ, seg_pos={0 0}, upload=56080]: Uploading piece... Jul 26 22:01:47.831987444 (58h25m57.554550504s) [noise=true , address=79.127.205.246:4276 , piece_id=PLDZKRBSNZLXWMH4GMJKQVTVLIKBYZ5MDRV6ZZBH65GLI5P6YHTQ, seg_pos={0 0}, upload=56080]: Uploading piece... Jul 26 22:01:47.832360458 (58h25m57.554923488s) [noise=true , address=104.36.82.6:4891 , piece_id=WVELUVBK6HQ3IPU6YMGKNGXV5WQKMYE3NUA75BMH5CWFQE342KYQ, seg_pos={0 0}, upload=56080]: Uploading piece... Jul 26 22:01:47.832624512 (58h25m57.555187542s) [noise=true , address=104.36.82.54:4164 , piece_id=FAD2DSXDQWL6CZYABQL5JSDXGF36VPT4Q3KBRIO5J5J4G7LE75GA, seg_pos={0 0}, upload=56080]: Uploading piece... Jul 26 22:01:47.832894699 (58h25m57.55545774s) [noise=true , address=104.36.82.38:4720 , piece_id=VNR3AUB5VUUL3L4DXAYE6AHPN42EX5YTF63XLQOOTRASRDUYHPMQ, seg_pos={0 0}, upload=56080]: Uploading piece... Jul 26 22:01:48.930398855 (58h25m58.652961895s) [upload=56080]: Next returned nil segment Jul 26 22:01:48.930421258 (58h25m58.652984298s) [upload=56080]: Waiting for error group managing segments... Jul 26 22:01:49.248162275 (58h25m58.970725316s) [noise=true , address=79.127.205.235:4662 , piece_id=QLODJQWXOVWV6LPYXTLVBQO7DY32FDXEQAODJIYAKSXDATEB4MGQ, seg_pos={0 0}, upload=56080]: Done uploading piece. err: <nil> Jul 26 22:01:49.316426436 (58h25m59.038989476s) [noise=true , address=79.127.205.234:4605 , piece_id=SP4QTS45Z5EJ7OOISC2QD4LGRQNBBZAKNMPQZNXFL5OACQ6UWKZQ, seg_pos={0 0}, upload=56080]: Done uploading piece. err: <nil> Jul 26 22:01:49.403186766 (58h25m59.125749796s) [noise=true , address=79.127.205.246:4276 , piece_id=PLDZKRBSNZLXWMH4GMJKQVTVLIKBYZ5MDRV6ZZBH65GLI5P6YHTQ, seg_pos={0 0}, upload=56080]: Done uploading piece. err: <nil> Jul 26 22:02:16.668232837 (58h26m26.390795907s) [noise=true , address=77.90.31.10:4531 , piece_id=QHOJQKHKAZAI6KP333OMH6BW2GA6SFMRT7NEJMIM4AFOCOAL7SFQ, seg_pos={0 0}, upload=56080]: Done uploading piece. err: <nil> Jul 26 22:02:30.538231207 (58h26m40.260794247s) [noise=true , address=77.90.31.18:4821 , piece_id=Y2T3ZSQOGMCTQVTKZVPP3LAXXTEBXIRRVCCEEMTXML6COOHMZ5WQ, seg_pos={0 0}, upload=56080]: Done uploading piece. err: <nil> Jul 26 22:02:46.737185635 (58h26m56.459748675s) [noise=true , address=104.36.82.38:4720 , piece_id=VNR3AUB5VUUL3L4DXAYE6AHPN42EX5YTF63XLQOOTRASRDUYHPMQ, seg_pos={0 0}, upload=56080]: Done uploading piece. err: ecclient: upload failed (node:12Y16Z5H7jpsCyRzsAajF5AR3mURqTY8a4jgEr2xP4Yxy9W9kdC, address:104.36.82.38:4720): protocol: rpc: dial tcp 104.36.82.38:4720: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:02:46.737238176 (58h26m56.459801226s) [noise=true , address=104.36.82.52:4100 , piece_id=WFL4OIJZTEHACRRD46IRJOXOYTHIULU7OBBJRAZB5R6ARDSR4LTA, seg_pos={0 0}, upload=56080]: Done uploading piece. err: ecclient: upload failed (node:12JQ78Lg28PborUyvkyLJb5RikJ8zJSAT5GkpYSsZLc6HRmkhyw, address:104.36.82.52:4100): protocol: rpc: dial tcp 104.36.82.52:4100: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:02:46.737272481 (58h26m56.459835521s) [noise=true , address=104.36.82.49:4979 , piece_id=FZAUGYJVNLXE4ENXIATPI6W4FCTOT7NXWFFOCSWZ2OYROLZCR2AQ, seg_pos={0 0}, upload=56080]: Done uploading piece. err: ecclient: upload failed (node:12B518TvbbRhj6LiXxV4GwUgAoUYmWMWeTpyYerKnKuZYBjXoma, address:104.36.82.49:4979): protocol: rpc: dial tcp 104.36.82.49:4979: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:02:46.737286759 (58h26m56.459849799s) [noise=true , address=104.36.82.54:4164 , piece_id=FAD2DSXDQWL6CZYABQL5JSDXGF36VPT4Q3KBRIO5J5J4G7LE75GA, seg_pos={0 0}, upload=56080]: Done uploading piece. err: ecclient: upload failed (node:1gUXWffNccBoXWftiMWHHSTeEpA5zrd2cHz4uXX5j75GeEK9s2, address:104.36.82.54:4164): protocol: rpc: dial tcp 104.36.82.54:4164: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:02:46.737324701 (58h26m56.459887741s) [noise=true , address=104.36.82.55:4200 , piece_id=OT7A57QX7R72N3OLAN2SMY6OKOXFM3S3G6YHQV5WVZUEQNRC2JPQ, seg_pos={0 0}, upload=56080]: Done uploading piece. err: ecclient: upload failed (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj, address:104.36.82.55:4200): protocol: rpc: dial tcp 104.36.82.55:4200: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:02:46.737358535 (58h26m56.459921576s) [noise=true , address=104.36.82.53:4145 , piece_id=FHZISQBV767WJUXATGQLZV3EWY3XP64VBWHINE6N63AELXSAD5HQ, seg_pos={0 0}, upload=56080]: Done uploading piece. err: ecclient: upload failed (node:1rXBNYi6SgZdwGbiU5ooBE8U7DrEoAGw25ncYsRETgNh4CsiLX, address:104.36.82.53:4145): protocol: rpc: dial tcp 104.36.82.53:4145: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:02:46.737365398 (58h26m56.459928448s) [noise=true , address=104.36.82.50:4030 , piece_id=MZBPVSEASYETAWOFJNANHHYR3WYJXORTA6SESN2E7FAJOOZXD77Q, seg_pos={0 0}, upload=56080]: Done uploading piece. err: ecclient: upload failed (node:1fRTqHVZiYV7NBAp7PLEPVXGEDZ7VXxQJuRhkXkAkNpT7ra27e, address:104.36.82.50:4030): protocol: rpc: dial tcp 104.36.82.50:4030: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:02:46.737396699 (58h26m56.459959739s) [noise=true , address=104.36.82.7:4942 , piece_id=U2S24U6FNNGGK6OB5U5LLWUKKRUIWNCVQVZ3HFM34LERDXWM4YHQ, seg_pos={0 0}, upload=56080]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:104.36.82.7:4942): protocol: rpc: dial tcp 104.36.82.7:4942: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:02:46.737404033 (58h26m56.459967073s) [noise=true , address=104.36.82.18:4390 , piece_id=P5CYDJ5KKWCAOYQFYDTFJHRNJ7B2VV56CG6SFR5N3NFYPTM62MJQ, seg_pos={0 0}, upload=56080]: Done uploading piece. err: ecclient: upload failed (node:12AZCJHCR7rF4r62Bg7NcBkpNoLb6r6rEN2a5hZ9KNxJTtjV8jp, address:104.36.82.18:4390): protocol: rpc: dial tcp 104.36.82.18:4390: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:02:46.737411707 (58h26m56.459974747s) [noise=true , address=104.36.82.47:4530 , piece_id=J4R26DQC5LPAZ4G2SUYWDIFK74PLBYR3N6L2CIBAXZNDXDB7QS6Q, seg_pos={0 0}, upload=56080]: Done uploading piece. err: ecclient: upload failed (node:1WVxMFx9jm6CT7GcXg2qgq3Be9UPFf1KZKssqufSk2ebdBBmxR, address:104.36.82.47:4530): protocol: rpc: dial tcp 104.36.82.47:4530: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:02:46.737434661 (58h26m56.459997701s) [noise=true , address=104.36.82.56:4231 , piece_id=ZQD3ZSQP2EFLRN2COOZGQ4NFCLDZ74UKGKNILGB2AUKZTL3H3LVQ, seg_pos={0 0}, upload=56080]: Done uploading piece. err: ecclient: upload failed (node:129WWyyn7SrJFbFJ2kjPVhwDNRoYM7Kj449TZiZLzUbqpwK7qJ4, address:104.36.82.56:4231): protocol: rpc: dial tcp 104.36.82.56:4231: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:02:46.737444610 (58h26m56.46000765s) [noise=true , address=104.36.82.46:4482 , piece_id=3FN3UKJGGNEZJRGAKHFAFHAKRUARCSIGDFEMKGOKEO4D2KCR2HGA, seg_pos={0 0}, upload=56080]: Done uploading piece. err: ecclient: upload failed (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, address:104.36.82.46:4482): protocol: rpc: dial tcp 104.36.82.46:4482: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:02:46.737458707 (58h26m56.460021748s) [noise=true , address=104.36.82.6:4891 , piece_id=WVELUVBK6HQ3IPU6YMGKNGXV5WQKMYE3NUA75BMH5CWFQE342KYQ, seg_pos={0 0}, upload=56080]: Done uploading piece. err: ecclient: upload failed (node:12Newezba9KBtr5PcBVDypYdAxRFdsMtVMUTMcPWLWPLziaJhyK, address:104.36.82.6:4891): protocol: write tcp 188.166.225.170:35258->104.36.82.6:4891: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:02:46.737486631 (58h26m56.460049671s) [seg_pos={0 0}, upload=56080]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil> Jul 26 22:02:46.967160518 (58h26m56.689723558s) [upload=56080]: Done uploading segments... Jul 26 22:02:46.967185806 (58h26m56.689748847s) [upload=56080]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5) Duration: 1m0.231966999s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [................X....X......XX....XX.XX.XXX.X.XX.XXX.XXXXXX.] duration: 15.06818415s FAIL (2025-07-26 22:02:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== CEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:42362->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:42362->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42362->77.90.31.10:4517: use of closed network connection Jul 26 22:01:56.285677843 (58h26m6.008240883s) [seg_pos={0 0}, upload=56076]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:01:56.513982741 (58h26m6.236545781s) [upload=56076]: Done uploading segments... Jul 26 22:01:56.514008370 (58h26m6.23657141s) [upload=56076]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 26 22:01:57.390344722 (58h26m7.112907762s) [upload=56082]: Starting upload Jul 26 22:01:57.390413313 (58h26m7.112976353s) [upload=56082]: Done starting upload Jul 26 22:01:57.390523063 (58h26m7.113086103s) [upload=56082]: Uploading segments... Jul 26 22:01:57.390607184 (58h26m7.113170224s) [upload=56082]: Got next segment. Inline: false Jul 26 22:01:57.390634677 (58h26m7.113197717s) [upload=56082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:01:57.390650066 (58h26m7.113213106s) [upload=56082]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:01:57.832734230 (58h26m7.55529727s) [seg_pos={0 0}, upload=56082]: Begin upload segment... Jul 26 22:01:57.832776451 (58h26m7.555339491s) [seg_pos={0 0}, upload=56082]: Done begin upload segment. Jul 26 22:01:57.832800907 (58h26m7.555363947s) [noise=true , address=45.91.116.13:4154 , piece_id=AISKJI4VJHWUGCUERMPZYAV2PVN5CLMK5FSW7MV3PAODNRDIZEUA, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:57.832834942 (58h26m7.555397982s) [noise=true , address=45.91.116.8:4045 , piece_id=TUZS6EVEZ62TJJ7KD2WJ45AJKYDC3U4X642ZXANXRXWXWNLBM3XA, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:57.833157620 (58h26m7.55572066s) [noise=true , address=45.91.116.12:4227 , piece_id=FFEJZEVNZHWCQ5XY7MAPVJDZJF4SBGTQNJZI6CTAQBYOJJWPQ7FA, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:57.833205873 (58h26m7.555768913s) [noise=true , address=45.91.116.9:4262 , piece_id=P64ZMXR2I5VBJQLD6IN3OBNMTZRXU22LJ5FZXCVD7XQFE62RPS6Q, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:57.833414842 (58h26m7.555977882s) [noise=true , address=77.90.31.10:4519 , piece_id=IGYIDV75DZSUT5QBCPYSL7OTDBOUN2CPAGPKA7LUI6OGG34ATEPQ, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:57.833504374 (58h26m7.556067404s) [noise=true , address=45.91.116.5:4298 , piece_id=BSLK2XR7HC236XWVMTZ67MDEEOD6C3XIXNFBXRQVUMCU234NYNKA, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:57.833656244 (58h26m7.556219284s) [noise=true , address=45.91.116.4:4938 , piece_id=7BXISFB73QQVKKSLQNDWR2OJI3CQJ3ZK3XMM3QCWQ3CDK4ZLY5IA, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:57.833826269 (58h26m7.556389309s) [noise=true , address=104.36.82.7:4909 , piece_id=E7N4ASRRZVVPF27Z2QTGESCI76SRYQHLL44LQLYH2GEFZGLYZAJQ, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:57.833922884 (58h26m7.556485924s) [noise=true , address=77.90.31.16:4735 , piece_id=YB5ZNU7M7D5FQZWIRO67B3VBTFDNQB4UIKTVFAIZ2J3GKRUOV7NQ, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:57.834167713 (58h26m7.556730753s) [noise=true , address=104.36.82.5:4837 , piece_id=53PAGMESJHS3IYYTAVGCYDBOHDRQLQPVYB3EB423DNJMI5FMZATQ, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:57.834230172 (58h26m7.556793212s) [noise=true , address=45.91.116.7:4009 , piece_id=6BSRDBUMTNALWY3KHN4HXIL3DJHC6PXAQASZ6DPS5CG6D5CPRMEA, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:57.834367896 (58h26m7.556930936s) [noise=true , address=104.36.82.16:4293 , piece_id=QZGF3ASFDALJCFCT7OYDJBN5GYCKNGVKHKQE277ZU62RW3Q2TZSA, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:57.834472486 (58h26m7.557035526s) [noise=true , address=104.36.82.15:4258 , piece_id=CMRIEATD2Q3Y7EDSH4ITUVCI7XWRS4RKTUGID4BJHBROJR3PGXMQ, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:57.834555004 (58h26m7.557118044s) [noise=true , address=77.90.31.12:4592 , piece_id=5CZXLU63WV2OJMMJBC3ALOKKKT7XGTDIUJGLL3B76HBFWV3DOASA, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:57.834703618 (58h26m7.557266648s) [noise=true , address=45.91.116.6:4974 , piece_id=NGESKWVBJ5O736MPOY5R53FHVMSRCXVEZQRCXUSZCOKBHNZJYOMQ, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:57.834745158 (58h26m7.557308198s) [noise=true , address=45.91.116.11:4117 , piece_id=PBEWSLZWLB6DYR2R4WYGOJPCLSA4KOUHVDLYKBGGHQBQGYELZ6ZQ, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:59.573575768 (58h26m9.296138808s) [upload=56082]: Next returned nil segment Jul 26 22:01:59.573593682 (58h26m9.296156712s) [upload=56082]: Waiting for error group managing segments... Jul 26 22:01:59.849382632 (58h26m9.571945672s) [noise=true , address=45.91.116.13:4154 , piece_id=AISKJI4VJHWUGCUERMPZYAV2PVN5CLMK5FSW7MV3PAODNRDIZEUA, seg_pos={0 0}, upload=56082]: Done uploading piece. err: <nil> Jul 26 22:01:59.859793060 (58h26m9.58235609s) [noise=true , address=45.91.116.8:4045 , piece_id=TUZS6EVEZ62TJJ7KD2WJ45AJKYDC3U4X642ZXANXRXWXWNLBM3XA, seg_pos={0 0}, upload=56082]: Done uploading piece. err: <nil> Jul 26 22:01:59.923504861 (58h26m9.646067901s) [noise=true , address=45.91.116.5:4298 , piece_id=BSLK2XR7HC236XWVMTZ67MDEEOD6C3XIXNFBXRQVUMCU234NYNKA, seg_pos={0 0}, upload=56082]: Done uploading piece. err: <nil> Jul 26 22:01:59.937899672 (58h26m9.660462712s) [noise=true , address=45.91.116.11:4117 , piece_id=PBEWSLZWLB6DYR2R4WYGOJPCLSA4KOUHVDLYKBGGHQBQGYELZ6ZQ, seg_pos={0 0}, upload=56082]: Done uploading piece. err: <nil> Jul 26 22:01:59.953884096 (58h26m9.676447136s) [noise=true , address=45.91.116.7:4009 , piece_id=6BSRDBUMTNALWY3KHN4HXIL3DJHC6PXAQASZ6DPS5CG6D5CPRMEA, seg_pos={0 0}, upload=56082]: Done uploading piece. err: <nil> Jul 26 22:02:00.027736248 (58h26m9.750299288s) [noise=true , address=45.91.116.4:4938 , piece_id=7BXISFB73QQVKKSLQNDWR2OJI3CQJ3ZK3XMM3QCWQ3CDK4ZLY5IA, seg_pos={0 0}, upload=56082]: Done uploading piece. err: <nil> Jul 26 22:02:00.034667830 (58h26m9.75723087s) [noise=true , address=45.91.116.9:4262 , piece_id=P64ZMXR2I5VBJQLD6IN3OBNMTZRXU22LJ5FZXCVD7XQFE62RPS6Q, seg_pos={0 0}, upload=56082]: Done uploading piece. err: <nil> Jul 26 22:02:00.082456705 (58h26m9.805019745s) [noise=true , address=45.91.116.12:4227 , piece_id=FFEJZEVNZHWCQ5XY7MAPVJDZJF4SBGTQNJZI6CTAQBYOJJWPQ7FA, seg_pos={0 0}, upload=56082]: Done uploading piece. err: <nil> Jul 26 22:02:00.089126276 (58h26m9.811689316s) [noise=true , address=45.91.116.6:4974 , piece_id=NGESKWVBJ5O736MPOY5R53FHVMSRCXVEZQRCXUSZCOKBHNZJYOMQ, seg_pos={0 0}, upload=56082]: Done uploading piece. err: <nil> Jul 26 22:02:05.325925773 (58h26m15.048488813s) [noise=true , address=77.90.31.10:4519 , piece_id=IGYIDV75DZSUT5QBCPYSL7OTDBOUN2CPAGPKA7LUI6OGG34ATEPQ, seg_pos={0 0}, upload=56082]: Done uploading piece. err: <nil> Jul 26 22:02:22.010826242 (58h26m31.733389282s) [noise=true , address=77.90.31.12:4592 , piece_id=5CZXLU63WV2OJMMJBC3ALOKKKT7XGTDIUJGLL3B76HBFWV3DOASA, seg_pos={0 0}, upload=56082]: Done uploading piece. err: <nil> Jul 26 22:02:26.743265063 (58h26m36.465828103s) [noise=true , address=104.36.82.15:4258 , piece_id=CMRIEATD2Q3Y7EDSH4ITUVCI7XWRS4RKTUGID4BJHBROJR3PGXMQ, seg_pos={0 0}, upload=56082]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:54148->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:02:26.743406403 (58h26m36.465969443s) [noise=true , address=104.36.82.16:4293 , piece_id=QZGF3ASFDALJCFCT7OYDJBN5GYCKNGVKHKQE277ZU62RW3Q2TZSA, seg_pos={0 0}, upload=56082]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:02:26.743464755 (58h26m36.466027795s) [noise=true , address=77.90.31.16:4735 , piece_id=YB5ZNU7M7D5FQZWIRO67B3VBTFDNQB4UIKTVFAIZ2J3GKRUOV7NQ, seg_pos={0 0}, upload=56082]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:02:26.743648857 (58h26m36.466211897s) [noise=true , address=104.36.82.7:4909 , piece_id=E7N4ASRRZVVPF27Z2QTGESCI76SRYQHLL44LQLYH2GEFZGLYZAJQ, seg_pos={0 0}, upload=56082]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 188.166.225.170:37230->104.36.82.7:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:02:26.743655089 (58h26m36.466218129s) [noise=true , address=104.36.82.5:4837 , piece_id=53PAGMESJHS3IYYTAVGCYDBOHDRQLQPVYB3EB423DNJMI5FMZATQ, seg_pos={0 0}, upload=56082]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 188.166.225.170:43472->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:02:26.743682802 (58h26m36.466245832s) [seg_pos={0 0}, upload=56082]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 22:02:26.983797456 (58h26m36.706360496s) [upload=56082]: Done uploading segments... Jul 26 22:02:26.983823416 (58h26m36.706386456s) [upload=56082]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 15.068204419s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X..X........] duration: 5.984623383s FAIL (2025-07-26 22:02:39 UTC) orbiter warning: context deadline exceeded Duration: 5.984646188s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.................X....X......XX....XX.XX.XXX.X.XX.XXX.XXXXXX] duration: 30.242260334s FAIL (2025-07-26 22:02:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== CEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:42362->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:42362->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42362->77.90.31.10:4517: use of closed network connection Jul 26 22:01:56.285677843 (58h26m6.008240883s) [seg_pos={0 0}, upload=56076]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:01:56.513982741 (58h26m6.236545781s) [upload=56076]: Done uploading segments... Jul 26 22:01:56.514008370 (58h26m6.23657141s) [upload=56076]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 26 22:01:57.390344722 (58h26m7.112907762s) [upload=56082]: Starting upload Jul 26 22:01:57.390413313 (58h26m7.112976353s) [upload=56082]: Done starting upload Jul 26 22:01:57.390523063 (58h26m7.113086103s) [upload=56082]: Uploading segments... Jul 26 22:01:57.390607184 (58h26m7.113170224s) [upload=56082]: Got next segment. Inline: false Jul 26 22:01:57.390634677 (58h26m7.113197717s) [upload=56082]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:01:57.390650066 (58h26m7.113213106s) [upload=56082]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:01:57.832734230 (58h26m7.55529727s) [seg_pos={0 0}, upload=56082]: Begin upload segment... Jul 26 22:01:57.832776451 (58h26m7.555339491s) [seg_pos={0 0}, upload=56082]: Done begin upload segment. Jul 26 22:01:57.832800907 (58h26m7.555363947s) [noise=true , address=45.91.116.13:4154 , piece_id=AISKJI4VJHWUGCUERMPZYAV2PVN5CLMK5FSW7MV3PAODNRDIZEUA, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:57.832834942 (58h26m7.555397982s) [noise=true , address=45.91.116.8:4045 , piece_id=TUZS6EVEZ62TJJ7KD2WJ45AJKYDC3U4X642ZXANXRXWXWNLBM3XA, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:57.833157620 (58h26m7.55572066s) [noise=true , address=45.91.116.12:4227 , piece_id=FFEJZEVNZHWCQ5XY7MAPVJDZJF4SBGTQNJZI6CTAQBYOJJWPQ7FA, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:57.833205873 (58h26m7.555768913s) [noise=true , address=45.91.116.9:4262 , piece_id=P64ZMXR2I5VBJQLD6IN3OBNMTZRXU22LJ5FZXCVD7XQFE62RPS6Q, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:57.833414842 (58h26m7.555977882s) [noise=true , address=77.90.31.10:4519 , piece_id=IGYIDV75DZSUT5QBCPYSL7OTDBOUN2CPAGPKA7LUI6OGG34ATEPQ, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:57.833504374 (58h26m7.556067404s) [noise=true , address=45.91.116.5:4298 , piece_id=BSLK2XR7HC236XWVMTZ67MDEEOD6C3XIXNFBXRQVUMCU234NYNKA, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:57.833656244 (58h26m7.556219284s) [noise=true , address=45.91.116.4:4938 , piece_id=7BXISFB73QQVKKSLQNDWR2OJI3CQJ3ZK3XMM3QCWQ3CDK4ZLY5IA, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:57.833826269 (58h26m7.556389309s) [noise=true , address=104.36.82.7:4909 , piece_id=E7N4ASRRZVVPF27Z2QTGESCI76SRYQHLL44LQLYH2GEFZGLYZAJQ, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:57.833922884 (58h26m7.556485924s) [noise=true , address=77.90.31.16:4735 , piece_id=YB5ZNU7M7D5FQZWIRO67B3VBTFDNQB4UIKTVFAIZ2J3GKRUOV7NQ, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:57.834167713 (58h26m7.556730753s) [noise=true , address=104.36.82.5:4837 , piece_id=53PAGMESJHS3IYYTAVGCYDBOHDRQLQPVYB3EB423DNJMI5FMZATQ, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:57.834230172 (58h26m7.556793212s) [noise=true , address=45.91.116.7:4009 , piece_id=6BSRDBUMTNALWY3KHN4HXIL3DJHC6PXAQASZ6DPS5CG6D5CPRMEA, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:57.834367896 (58h26m7.556930936s) [noise=true , address=104.36.82.16:4293 , piece_id=QZGF3ASFDALJCFCT7OYDJBN5GYCKNGVKHKQE277ZU62RW3Q2TZSA, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:57.834472486 (58h26m7.557035526s) [noise=true , address=104.36.82.15:4258 , piece_id=CMRIEATD2Q3Y7EDSH4ITUVCI7XWRS4RKTUGID4BJHBROJR3PGXMQ, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:57.834555004 (58h26m7.557118044s) [noise=true , address=77.90.31.12:4592 , piece_id=5CZXLU63WV2OJMMJBC3ALOKKKT7XGTDIUJGLL3B76HBFWV3DOASA, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:57.834703618 (58h26m7.557266648s) [noise=true , address=45.91.116.6:4974 , piece_id=NGESKWVBJ5O736MPOY5R53FHVMSRCXVEZQRCXUSZCOKBHNZJYOMQ, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:57.834745158 (58h26m7.557308198s) [noise=true , address=45.91.116.11:4117 , piece_id=PBEWSLZWLB6DYR2R4WYGOJPCLSA4KOUHVDLYKBGGHQBQGYELZ6ZQ, seg_pos={0 0}, upload=56082]: Uploading piece... Jul 26 22:01:59.573575768 (58h26m9.296138808s) [upload=56082]: Next returned nil segment Jul 26 22:01:59.573593682 (58h26m9.296156712s) [upload=56082]: Waiting for error group managing segments... Jul 26 22:01:59.849382632 (58h26m9.571945672s) [noise=true , address=45.91.116.13:4154 , piece_id=AISKJI4VJHWUGCUERMPZYAV2PVN5CLMK5FSW7MV3PAODNRDIZEUA, seg_pos={0 0}, upload=56082]: Done uploading piece. err: <nil> Jul 26 22:01:59.859793060 (58h26m9.58235609s) [noise=true , address=45.91.116.8:4045 , piece_id=TUZS6EVEZ62TJJ7KD2WJ45AJKYDC3U4X642ZXANXRXWXWNLBM3XA, seg_pos={0 0}, upload=56082]: Done uploading piece. err: <nil> Jul 26 22:01:59.923504861 (58h26m9.646067901s) [noise=true , address=45.91.116.5:4298 , piece_id=BSLK2XR7HC236XWVMTZ67MDEEOD6C3XIXNFBXRQVUMCU234NYNKA, seg_pos={0 0}, upload=56082]: Done uploading piece. err: <nil> Jul 26 22:01:59.937899672 (58h26m9.660462712s) [noise=true , address=45.91.116.11:4117 , piece_id=PBEWSLZWLB6DYR2R4WYGOJPCLSA4KOUHVDLYKBGGHQBQGYELZ6ZQ, seg_pos={0 0}, upload=56082]: Done uploading piece. err: <nil> Jul 26 22:01:59.953884096 (58h26m9.676447136s) [noise=true , address=45.91.116.7:4009 , piece_id=6BSRDBUMTNALWY3KHN4HXIL3DJHC6PXAQASZ6DPS5CG6D5CPRMEA, seg_pos={0 0}, upload=56082]: Done uploading piece. err: <nil> Jul 26 22:02:00.027736248 (58h26m9.750299288s) [noise=true , address=45.91.116.4:4938 , piece_id=7BXISFB73QQVKKSLQNDWR2OJI3CQJ3ZK3XMM3QCWQ3CDK4ZLY5IA, seg_pos={0 0}, upload=56082]: Done uploading piece. err: <nil> Jul 26 22:02:00.034667830 (58h26m9.75723087s) [noise=true , address=45.91.116.9:4262 , piece_id=P64ZMXR2I5VBJQLD6IN3OBNMTZRXU22LJ5FZXCVD7XQFE62RPS6Q, seg_pos={0 0}, upload=56082]: Done uploading piece. err: <nil> Jul 26 22:02:00.082456705 (58h26m9.805019745s) [noise=true , address=45.91.116.12:4227 , piece_id=FFEJZEVNZHWCQ5XY7MAPVJDZJF4SBGTQNJZI6CTAQBYOJJWPQ7FA, seg_pos={0 0}, upload=56082]: Done uploading piece. err: <nil> Jul 26 22:02:00.089126276 (58h26m9.811689316s) [noise=true , address=45.91.116.6:4974 , piece_id=NGESKWVBJ5O736MPOY5R53FHVMSRCXVEZQRCXUSZCOKBHNZJYOMQ, seg_pos={0 0}, upload=56082]: Done uploading piece. err: <nil> Jul 26 22:02:05.325925773 (58h26m15.048488813s) [noise=true , address=77.90.31.10:4519 , piece_id=IGYIDV75DZSUT5QBCPYSL7OTDBOUN2CPAGPKA7LUI6OGG34ATEPQ, seg_pos={0 0}, upload=56082]: Done uploading piece. err: <nil> Jul 26 22:02:22.010826242 (58h26m31.733389282s) [noise=true , address=77.90.31.12:4592 , piece_id=5CZXLU63WV2OJMMJBC3ALOKKKT7XGTDIUJGLL3B76HBFWV3DOASA, seg_pos={0 0}, upload=56082]: Done uploading piece. err: <nil> Jul 26 22:02:26.743265063 (58h26m36.465828103s) [noise=true , address=104.36.82.15:4258 , piece_id=CMRIEATD2Q3Y7EDSH4ITUVCI7XWRS4RKTUGID4BJHBROJR3PGXMQ, seg_pos={0 0}, upload=56082]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:54148->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:02:26.743406403 (58h26m36.465969443s) [noise=true , address=104.36.82.16:4293 , piece_id=QZGF3ASFDALJCFCT7OYDJBN5GYCKNGVKHKQE277ZU62RW3Q2TZSA, seg_pos={0 0}, upload=56082]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:02:26.743464755 (58h26m36.466027795s) [noise=true , address=77.90.31.16:4735 , piece_id=YB5ZNU7M7D5FQZWIRO67B3VBTFDNQB4UIKTVFAIZ2J3GKRUOV7NQ, seg_pos={0 0}, upload=56082]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:02:26.743648857 (58h26m36.466211897s) [noise=true , address=104.36.82.7:4909 , piece_id=E7N4ASRRZVVPF27Z2QTGESCI76SRYQHLL44LQLYH2GEFZGLYZAJQ, seg_pos={0 0}, upload=56082]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: write tcp 188.166.225.170:37230->104.36.82.7:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:02:26.743655089 (58h26m36.466218129s) [noise=true , address=104.36.82.5:4837 , piece_id=53PAGMESJHS3IYYTAVGCYDBOHDRQLQPVYB3EB423DNJMI5FMZATQ, seg_pos={0 0}, upload=56082]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 188.166.225.170:43472->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:02:26.743682802 (58h26m36.466245832s) [seg_pos={0 0}, upload=56082]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 22:02:26.983797456 (58h26m36.706360496s) [upload=56082]: Done uploading segments... Jul 26 22:02:26.983823416 (58h26m36.706386456s) [upload=56082]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.242276886s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 21 out of 60 is failed (threshold=3) [........................XX.........XX.XX.XXXX.XX.XXXXX.X.XXX] duration: 27.667791718s FAIL (2025-07-26 22:02:25 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/4M Duration: 27.667820935s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [X.................X.........................................] duration: 2.223094558s FAIL (2025-07-26 22:01:59 UTC) orbiter warning: digest mismatch: after 0783c6a051cf3b535b7c61f8e6f0cfe751970c7121d9bce2182fd51e29ca8fc6 != before ee59801cb45aa958945ed73fcee20d9c3604e7f558c75f49df119aca36a29d88 Duration: 2.223122502s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..................X....X......XX....XX.XX.XXX.X.XX.XXX.XXXXX] duration: 30.230249653s FAIL (2025-07-26 22:01:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=F4BIRUOTBA4IV5WZFU7YUIYU6YO3G7I6STK5VDYL7XKX4BR5FCBQ, seg_pos={0 0}, upload=56069]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:55504->104.36.82.16:4295: use of closed network connection; conn closed: op Jul 26 22:01:25.843191757 (58h25m35.565754797s) [seg_pos={0 0}, upload=56069]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:01:26.072871977 (58h25m35.795435027s) [upload=56069]: Done uploading segments... Jul 26 22:01:26.072902745 (58h25m35.795465785s) [upload=56069]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 26 22:01:26.936807242 (58h25m36.659370282s) [upload=56076]: Starting upload Jul 26 22:01:26.936887656 (58h25m36.659450696s) [upload=56076]: Done starting upload Jul 26 22:01:26.937017083 (58h25m36.659580123s) [upload=56076]: Uploading segments... Jul 26 22:01:26.937187029 (58h25m36.659750069s) [upload=56076]: Got next segment. Inline: false Jul 26 22:01:26.937233988 (58h25m36.659797028s) [upload=56076]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:01:26.937239319 (58h25m36.659802359s) [upload=56076]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:01:27.175061666 (58h25m36.897624706s) [seg_pos={0 0}, upload=56076]: Begin upload segment... Jul 26 22:01:27.175110239 (58h25m36.897673279s) [seg_pos={0 0}, upload=56076]: Done begin upload segment. Jul 26 22:01:27.175134666 (58h25m36.897697706s) [noise=true , address=77.90.31.16:4734 , piece_id=ERDH5GTG2462ATKAZJLCHG5FZDQEPSKMAAML77X7OB2CMJNCYD3Q, seg_pos={0 0}, upload=56076]: Uploading piece... Jul 26 22:01:27.175185373 (58h25m36.897748413s) [noise=true , address=45.91.116.11:4119 , piece_id=FUWJPBYWIG3XONCUTDKGFQYRSAJRU565GP7TLV56O6KAHFUQEZSA, seg_pos={0 0}, upload=56076]: Uploading piece... Jul 26 22:01:27.175512899 (58h25m36.898075929s) [noise=true , address=45.91.116.14:4190 , piece_id=WJE5QEPGATX5E2D7OVDQ4TMHC3YX3AIUNUBZT3XSMQ4QPCBNYK3Q, seg_pos={0 0}, upload=56076]: Uploading piece... Jul 26 22:01:27.175716008 (58h25m36.898279088s) [noise=true , address=45.91.116.6:4973 , piece_id=QIAJYHCW7LC5K7YNODMLWTXPKTXATX5QDN7A6OAYWI72QT47G5UA, seg_pos={0 0}, upload=56076]: Uploading piece... Jul 26 22:01:27.175837931 (58h25m36.898400971s) [noise=true , address=77.90.31.12:4589 , piece_id=UESAY2RCL7UWBUEBMGBELVX6ITJKZEHYZQVXO36R3XZOHUWIAXNA, seg_pos={0 0}, upload=56076]: Uploading piece... Jul 26 22:01:27.176092779 (58h25m36.898655859s) [noise=true , address=104.36.82.9:4005 , piece_id=YN6G56ZMPETDX44D6NNDNOZURM6IADCSCFSQMFR2DSFEZBDV6XUA, seg_pos={0 0}, upload=56076]: Uploading piece... Jul 26 22:01:27.176142023 (58h25m36.898705063s) [noise=true , address=77.90.31.10:4517 , piece_id=GYFYZFBERMOBGAPVHK6HRGP4ZH4PWEU4TZW2CHMRCCSUMAN7KQDQ, seg_pos={0 0}, upload=56076]: Uploading piece... Jul 26 22:01:27.176467826 (58h25m36.899030866s) [noise=true , address=45.91.116.5:4299 , piece_id=VL4L5JJQ23P2BFO5OWZRDE6H65N7ZTPCXQSV3KYAYWSLMMIGULHQ, seg_pos={0 0}, upload=56076]: Uploading piece... Jul 26 22:01:27.176614478 (58h25m36.899177518s) [noise=true , address=104.36.82.13:4187 , piece_id=TJ5FAEWJ2DKQGXPQ2MHRSRANMVSNWUVLMCT3HM2SKNO3DGI4M6RA, seg_pos={0 0}, upload=56076]: Uploading piece... Jul 26 22:01:27.176703668 (58h25m36.899266708s) [noise=true , address=45.91.116.13:4155 , piece_id=FUCAPEKWP5AYBGCLNDVTTSO7CYKXDE7U367V7YQITNYH3FLTIN7Q, seg_pos={0 0}, upload=56076]: Uploading piece... Jul 26 22:01:27.176938837 (58h25m36.899501877s) [noise=true , address=45.91.116.9:4263 , piece_id=RLZPVRQVFPRFBDPSIDCVPHCBCPVWWETGFMVLZZQZEJKA4OPNM4DQ, seg_pos={0 0}, upload=56076]: Uploading piece... Jul 26 22:01:27.177194086 (58h25m36.899757126s) [noise=true , address=77.90.31.14:4663 , piece_id=26EX6BEYSWSDFK2VHUJYFRC6AZJ3TVJKRNES2DQHIZNE5KJXYGBQ, seg_pos={0 0}, upload=56076]: Uploading piece... Jul 26 22:01:27.177411673 (58h25m36.899974713s) [noise=true , address=77.90.31.15:4699 , piece_id=VARXM7772H2FDMLJ5K33NWPRNGXMW3KMIJFNSPJGJBQHLWD6IERQ, seg_pos={0 0}, upload=56076]: Uploading piece... Jul 26 22:01:27.177623487 (58h25m36.900186527s) [noise=true , address=77.90.31.16:4735 , piece_id=I42QNAR6Z46LJXBFNPTHBX3O26RKAW6LM5EYSSZG7YBUENOWYADA, seg_pos={0 0}, upload=56076]: Uploading piece... Jul 26 22:01:27.177864098 (58h25m36.900427138s) [noise=true , address=45.91.116.12:4226 , piece_id=LUAC2D7P2B66PA4BJUKMOE7EMMTHA6YZR6UWQ2IK5K3EIIOY7CAQ, seg_pos={0 0}, upload=56076]: Uploading piece... Jul 26 22:01:27.178096612 (58h25m36.900659652s) [noise=true , address=45.91.116.10:4082 , piece_id=D7JMRB3NBMNP3BPB4LWVBGO7OLMCD4XRQ27ASIYDSDS65N4M267Q, seg_pos={0 0}, upload=56076]: Uploading piece... Jul 26 22:01:28.934314141 (58h25m38.656877181s) [upload=56076]: Next returned nil segment Jul 26 22:01:28.934332166 (58h25m38.656895206s) [upload=56076]: Waiting for error group managing segments... Jul 26 22:01:29.211212423 (58h25m38.933775463s) [noise=true , address=45.91.116.9:4263 , piece_id=RLZPVRQVFPRFBDPSIDCVPHCBCPVWWETGFMVLZZQZEJKA4OPNM4DQ, seg_pos={0 0}, upload=56076]: Done uploading piece. err: <nil> Jul 26 22:01:29.277695847 (58h25m39.000258887s) [noise=true , address=45.91.116.6:4973 , piece_id=QIAJYHCW7LC5K7YNODMLWTXPKTXATX5QDN7A6OAYWI72QT47G5UA, seg_pos={0 0}, upload=56076]: Done uploading piece. err: <nil> Jul 26 22:01:29.278225380 (58h25m39.00078842s) [noise=true , address=45.91.116.12:4226 , piece_id=LUAC2D7P2B66PA4BJUKMOE7EMMTHA6YZR6UWQ2IK5K3EIIOY7CAQ, seg_pos={0 0}, upload=56076]: Done uploading piece. err: <nil> Jul 26 22:01:29.285167251 (58h25m39.007730291s) [noise=true , address=45.91.116.11:4119 , piece_id=FUWJPBYWIG3XONCUTDKGFQYRSAJRU565GP7TLV56O6KAHFUQEZSA, seg_pos={0 0}, upload=56076]: Done uploading piece. err: <nil> Jul 26 22:01:29.310300289 (58h25m39.032863329s) [noise=true , address=45.91.116.10:4082 , piece_id=D7JMRB3NBMNP3BPB4LWVBGO7OLMCD4XRQ27ASIYDSDS65N4M267Q, seg_pos={0 0}, upload=56076]: Done uploading piece. err: <nil> Jul 26 22:01:29.348920622 (58h25m39.071483662s) [noise=true , address=45.91.116.14:4190 , piece_id=WJE5QEPGATX5E2D7OVDQ4TMHC3YX3AIUNUBZT3XSMQ4QPCBNYK3Q, seg_pos={0 0}, upload=56076]: Done uploading piece. err: <nil> Jul 26 22:01:29.358455175 (58h25m39.081018215s) [noise=true , address=45.91.116.5:4299 , piece_id=VL4L5JJQ23P2BFO5OWZRDE6H65N7ZTPCXQSV3KYAYWSLMMIGULHQ, seg_pos={0 0}, upload=56076]: Done uploading piece. err: <nil> Jul 26 22:01:29.361337469 (58h25m39.08390051s) [noise=true , address=45.91.116.13:4155 , piece_id=FUCAPEKWP5AYBGCLNDVTTSO7CYKXDE7U367V7YQITNYH3FLTIN7Q, seg_pos={0 0}, upload=56076]: Done uploading piece. err: <nil> Jul 26 22:01:35.821407315 (58h25m45.543970355s) [noise=true , address=77.90.31.16:4734 , piece_id=ERDH5GTG2462ATKAZJLCHG5FZDQEPSKMAAML77X7OB2CMJNCYD3Q, seg_pos={0 0}, upload=56076]: Done uploading piece. err: <nil> Jul 26 22:01:51.426425236 (58h26m1.148988276s) [noise=true , address=77.90.31.15:4699 , piece_id=VARXM7772H2FDMLJ5K33NWPRNGXMW3KMIJFNSPJGJBQHLWD6IERQ, seg_pos={0 0}, upload=56076]: Done uploading piece. err: <nil> Jul 26 22:01:54.324535389 (58h26m4.047098429s) [noise=true , address=77.90.31.16:4735 , piece_id=I42QNAR6Z46LJXBFNPTHBX3O26RKAW6LM5EYSSZG7YBUENOWYADA, seg_pos={0 0}, upload=56076]: Done uploading piece. err: <nil> Jul 26 22:01:55.268148554 (58h26m4.990711594s) [noise=true , address=77.90.31.12:4589 , piece_id=UESAY2RCL7UWBUEBMGBELVX6ITJKZEHYZQVXO36R3XZOHUWIAXNA, seg_pos={0 0}, upload=56076]: Done uploading piece. err: <nil> Jul 26 22:01:56.285315359 (58h26m6.007878409s) [noise=true , address=77.90.31.14:4663 , piece_id=26EX6BEYSWSDFK2VHUJYFRC6AZJ3TVJKRNES2DQHIZNE5KJXYGBQ, seg_pos={0 0}, upload=56076]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:36184->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:36184->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36184->77.90.31.14:4663: use of closed network connection Jul 26 22:01:56.285413396 (58h26m6.007976436s) [noise=true , address=104.36.82.13:4187 , piece_id=TJ5FAEWJ2DKQGXPQ2MHRSRANMVSNWUVLMCT3HM2SKNO3DGI4M6RA, seg_pos={0 0}, upload=56076]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:01:56.285596587 (58h26m6.008159627s) [noise=true , address=104.36.82.9:4005 , piece_id=YN6G56ZMPETDX44D6NNDNOZURM6IADCSCFSQMFR2DSFEZBDV6XUA, seg_pos={0 0}, upload=56076]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:01:56.285663716 (58h26m6.008226746s) [noise=true , address=77.90.31.10:4517 , piece_id=GYFYZFBERMOBGAPVHK6HRGP4ZH4PWEU4TZW2CHMRCCSUMAN7KQDQ, seg_pos={0 0}, upload=56076]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:42362->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:42362->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42362->77.90.31.10:4517: use of closed network connection Jul 26 22:01:56.285677843 (58h26m6.008240883s) [seg_pos={0 0}, upload=56076]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:01:56.513982741 (58h26m6.236545781s) [upload=56076]: Done uploading segments... Jul 26 22:01:56.514008370 (58h26m6.23657141s) [upload=56076]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.23026402s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 20 out of 60 is failed (threshold=3) [.........................XX.........XX.XX.XXXX.XX.XXXXX.X.XX] duration: 30.240294938s FAIL (2025-07-26 22:01:44 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.240312821s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.X.................X........................................] duration: 1.924083017s FAIL (2025-07-26 22:01:43 UTC) orbiter warning: digest mismatch: after 0783c6a051cf3b535b7c61f8e6f0cfe751970c7121d9bce2182fd51e29ca8fc6 != before ee59801cb45aa958945ed73fcee20d9c3604e7f558c75f49df119aca36a29d88 Duration: 1.924113154s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X..X.......] duration: 10.81901695s FAIL (2025-07-26 22:01:34 UTC) orbiter warning: context deadline exceeded Duration: 10.819038501s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 32 out of 60 is failed (threshold=3) [............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.234168577s FAIL (2025-07-26 22:01:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 5) ======LOGS====== 4434 , piece_id=25DKIAP3ZPS3ZS7K6Q4UNDUJOKZPWXOARLXMRJWYH2AGCN7M3U7Q, seg_pos={0 0}, upload=56059]: Uploading piece... Jul 26 22:00:34.289441740 (58h24m44.01200478s) [noise=true , address=104.36.82.39:4755 , piece_id=HOHCZLCCU6NZSQMYLW6S5S5RBBTOHVUYV56JBULTBWREXK2PHI6Q, seg_pos={0 0}, upload=56059]: Uploading piece... Jul 26 22:00:34.289676208 (58h24m44.012239248s) [noise=true , address=79.127.201.215:4352 , piece_id=KERR772BYZ46B5SHEBGU7W3FXBEHBPFI4YPEGU6IJZFHI534PQOQ, seg_pos={0 0}, upload=56059]: Uploading piece... Jul 26 22:00:34.289911899 (58h24m44.012474949s) [noise=true , address=104.36.82.52:4112 , piece_id=6UPNBBNCBQETPJNNZ3G6GAURNA34CPCXXTWOXPOA6IVHLGGNIFQQ, seg_pos={0 0}, upload=56059]: Uploading piece... Jul 26 22:00:34.290122382 (58h24m44.012685422s) [noise=true , address=104.36.82.53:4154 , piece_id=TKVUBXNVJEAIW4GGNCZL5CNALM2PIYN3EQE7SOE7XWOAY6RR36KQ, seg_pos={0 0}, upload=56059]: Uploading piece... Jul 26 22:00:34.290328376 (58h24m44.012891416s) [noise=true , address=104.36.82.35:4593 , piece_id=G5UPQC4APDVNPH2EKDH57HYFXHWATI5GBJZBHASQTEGO2CEDV4FQ, seg_pos={0 0}, upload=56059]: Uploading piece... Jul 26 22:00:34.290541925 (58h24m44.013104965s) [noise=true , address=104.36.82.55:4209 , piece_id=K2MVXYATRCWO7CVZUHK2CPMIM3P5ZQKLF6UT3WWQ3DQGFE463MTQ, seg_pos={0 0}, upload=56059]: Uploading piece... Jul 26 22:00:34.290765542 (58h24m44.013328572s) [noise=true , address=104.36.82.47:4535 , piece_id=W6NBPL52G4PUI4ANETLFKYCAPAFYQ3CQ5EBWMEHROQ6WEF5UBDJA, seg_pos={0 0}, upload=56059]: Uploading piece... Jul 26 22:00:34.290977929 (58h24m44.013540969s) [noise=true , address=104.36.82.56:4253 , piece_id=MRINAJKPOKZ4JZ5764XXOSHEGRPBZHXZYK2F2T36X73GH4X3DY5Q, seg_pos={0 0}, upload=56059]: Uploading piece... Jul 26 22:00:34.291190585 (58h24m44.013753625s) [noise=true , address=77.90.31.19:4867 , piece_id=HVC4SGGZ4M4Y67O3D4KCPSVUKFPITQKW6CJKJUQHMPBENBUUCSUA, seg_pos={0 0}, upload=56059]: Uploading piece... Jul 26 22:00:34.291387443 (58h24m44.013950483s) [noise=true , address=104.36.82.47:4547 , piece_id=E4U4NTM6USK3AFTW3HSAVS65WWH4TO5LWD2JCHJZHEUBQBODJLLQ, seg_pos={0 0}, upload=56059]: Uploading piece... Jul 26 22:00:34.291598075 (58h24m44.014161115s) [noise=true , address=104.36.82.10:4081 , piece_id=TB43RAESK6IIKJJR5ZGNHNAP2KSWRFAKFREE64UMA4AFXLV554TQ, seg_pos={0 0}, upload=56059]: Uploading piece... Jul 26 22:00:34.291804470 (58h24m44.01436751s) [noise=true , address=79.127.205.246:4281 , piece_id=4AJUCHF222CIWL3FOMB2SCFTPY6RAPZXIRHPRJ6K7LKTGMJB26IQ, seg_pos={0 0}, upload=56059]: Uploading piece... Jul 26 22:00:35.425324934 (58h24m45.147887974s) [upload=56059]: Next returned nil segment Jul 26 22:00:35.425342669 (58h24m45.147905709s) [upload=56059]: Waiting for error group managing segments... Jul 26 22:00:35.741314340 (58h24m45.46387738s) [noise=true , address=79.127.205.246:4281 , piece_id=4AJUCHF222CIWL3FOMB2SCFTPY6RAPZXIRHPRJ6K7LKTGMJB26IQ, seg_pos={0 0}, upload=56059]: Done uploading piece. err: <nil> Jul 26 22:00:36.356051322 (58h24m46.078614362s) [noise=true , address=156.146.43.225:4434 , piece_id=25DKIAP3ZPS3ZS7K6Q4UNDUJOKZPWXOARLXMRJWYH2AGCN7M3U7Q, seg_pos={0 0}, upload=56059]: Done uploading piece. err: <nil> Jul 26 22:00:36.649995287 (58h24m46.372558327s) [noise=true , address=79.127.219.38:4018 , piece_id=F36HZCKHBHEN76OZAZW7KX6TMOCVSLA6QKI227Y22H2NJIARCEZQ, seg_pos={0 0}, upload=56059]: Done uploading piece. err: <nil> Jul 26 22:00:42.414724190 (58h24m52.13728723s) [noise=true , address=79.127.201.215:4352 , piece_id=KERR772BYZ46B5SHEBGU7W3FXBEHBPFI4YPEGU6IJZFHI534PQOQ, seg_pos={0 0}, upload=56059]: Done uploading piece. err: <nil> Jul 26 22:00:52.524793948 (58h25m2.247356988s) [noise=true , address=77.90.31.19:4867 , piece_id=HVC4SGGZ4M4Y67O3D4KCPSVUKFPITQKW6CJKJUQHMPBENBUUCSUA, seg_pos={0 0}, upload=56059]: Done uploading piece. err: <nil> Jul 26 22:01:33.161012597 (58h25m42.883575637s) [noise=true , address=104.36.82.42:4831 , piece_id=GOVGCI2RARZG36XWM3JRTILMTBSXZVQR73CK2NT5JS6LNTXEGY2A, seg_pos={0 0}, upload=56059]: Done uploading piece. err: ecclient: upload failed (node:12hZK8xSEF54FaK2tBkB4BVHywDuJs45HisTs2SJ7ZrnuKkoDDX, address:104.36.82.42:4831): protocol: rpc: dial tcp 104.36.82.42:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:01:33.161352817 (58h25m42.883915857s) [noise=true , address=104.36.82.53:4154 , piece_id=TKVUBXNVJEAIW4GGNCZL5CNALM2PIYN3EQE7SOE7XWOAY6RR36KQ, seg_pos={0 0}, upload=56059]: Done uploading piece. err: ecclient: upload failed (node:1vbGunXCRMkmfvBqHPmbuW7fBLKck4Vh9MGsX1q7P6VfX6DKqB, address:104.36.82.53:4154): protocol: rpc: dial tcp 104.36.82.53:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:01:33.161394638 (58h25m42.883957668s) [noise=true , address=104.36.82.35:4593 , piece_id=G5UPQC4APDVNPH2EKDH57HYFXHWATI5GBJZBHASQTEGO2CEDV4FQ, seg_pos={0 0}, upload=56059]: Done uploading piece. err: ecclient: upload failed (node:12Q7uKA1U2EqaMAdCJ9ADpVj9XB9StQwxpdpVkUvrPHgmcoiHxg, address:104.36.82.35:4593): protocol: rpc: dial tcp 104.36.82.35:4593: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:01:33.161424344 (58h25m42.883987385s) [noise=true , address=104.36.82.9:4010 , piece_id=6BFG3X6Y6K773XPR2J2YD7JSSGLPRBTMVKPW2CPRT7KAI5DSPDFQ, seg_pos={0 0}, upload=56059]: Done uploading piece. err: ecclient: upload failed (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp, address:104.36.82.9:4010): protocol: rpc: dial tcp 104.36.82.9:4010: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:01:33.161458900 (58h25m42.88402194s) [noise=true , address=104.36.82.39:4755 , piece_id=HOHCZLCCU6NZSQMYLW6S5S5RBBTOHVUYV56JBULTBWREXK2PHI6Q, seg_pos={0 0}, upload=56059]: Done uploading piece. err: ecclient: upload failed (node:1cmNqubJ82X6bTLTn5zmMD7HhVDKyP8SH9vo2ZYq5S6GmX6Wpx, address:104.36.82.39:4755): protocol: rpc: dial tcp 104.36.82.39:4755: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:01:33.161488096 (58h25m42.884051136s) [noise=true , address=104.36.82.52:4112 , piece_id=6UPNBBNCBQETPJNNZ3G6GAURNA34CPCXXTWOXPOA6IVHLGGNIFQQ, seg_pos={0 0}, upload=56059]: Done uploading piece. err: ecclient: upload failed (node:1dQFyMxQ1vVeHWCmyUWFUtJNBTdRUP2bbdLJQCMwF9z1L1eapD, address:104.36.82.52:4112): protocol: rpc: dial tcp 104.36.82.52:4112: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:01:33.161553781 (58h25m42.884116821s) [noise=true , address=104.36.82.10:4081 , piece_id=TB43RAESK6IIKJJR5ZGNHNAP2KSWRFAKFREE64UMA4AFXLV554TQ, seg_pos={0 0}, upload=56059]: Done uploading piece. err: ecclient: upload failed (node:12t9rGfCQ7bLkNnZzaPVgkE2Y9rXsiFp1XGdeECzv9M1jk8Qqmp, address:104.36.82.10:4081): protocol: rpc: dial tcp 104.36.82.10:4081: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:01:33.161592686 (58h25m42.884155727s) [noise=true , address=104.36.82.47:4547 , piece_id=E4U4NTM6USK3AFTW3HSAVS65WWH4TO5LWD2JCHJZHEUBQBODJLLQ, seg_pos={0 0}, upload=56059]: Done uploading piece. err: ecclient: upload failed (node:12brQg9WQMKJgp1buz1wMS8W11XEwAx7RHG5h51EmLvwY5D5Hky, address:104.36.82.47:4547): protocol: rpc: dial tcp 104.36.82.47:4547: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:01:33.161613246 (58h25m42.884176286s) [noise=true , address=104.36.82.55:4209 , piece_id=K2MVXYATRCWO7CVZUHK2CPMIM3P5ZQKLF6UT3WWQ3DQGFE463MTQ, seg_pos={0 0}, upload=56059]: Done uploading piece. err: ecclient: upload failed (node:12EmE2BgEZTG1LTsZ5p9ghqDSfg5ZQATt31Tqg5jZ5KQE2UjDnR, address:104.36.82.55:4209): protocol: rpc: dial tcp 104.36.82.55:4209: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:01:33.161624788 (58h25m42.884187828s) [noise=true , address=104.36.82.56:4253 , piece_id=MRINAJKPOKZ4JZ5764XXOSHEGRPBZHXZYK2F2T36X73GH4X3DY5Q, seg_pos={0 0}, upload=56059]: Done uploading piece. err: ecclient: upload failed (node:1Pz6ZrFzAdLySCxtFhbotfPLiA4nSmvhp5uaRAJQgSc7DUgmap, address:104.36.82.56:4253): protocol: rpc: dial tcp 104.36.82.56:4253: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:01:33.161649355 (58h25m42.884212395s) [noise=true , address=104.36.82.47:4535 , piece_id=W6NBPL52G4PUI4ANETLFKYCAPAFYQ3CQ5EBWMEHROQ6WEF5UBDJA, seg_pos={0 0}, upload=56059]: Done uploading piece. err: ecclient: upload failed (node:12KFwCcqBCTyeBBQYzg6SFG4YDzBm2H9s4D2JRvnC31VAGvpLhu, address:104.36.82.47:4535): protocol: rpc: dial tcp 104.36.82.47:4535: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:01:33.161659564 (58h25m42.884222604s) [noise=true , address=104.36.82.41:4778 , piece_id=C3VCDIBSGZZ4KPCZKZA23R6LUBIV4J7M7RJTYWJTCO35OWNXACGQ, seg_pos={0 0}, upload=56059]: Done uploading piece. err: ecclient: upload failed (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV, address:104.36.82.41:4778): protocol: rpc: dial tcp 104.36.82.41:4778: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:01:33.161678370 (58h25m42.88424141s) [noise=true , address=104.36.82.10:4101 , piece_id=HMUXBNR67FIQ6DXJOCOAXVP4EEHBPMNDL6NZWNPMGQVJMTQRDEBA, seg_pos={0 0}, upload=56059]: Done uploading piece. err: ecclient: upload failed (node:1RmTDQrWoQBSpHrcUsb2PYijpcFdCyRya7VrMkxhWeQXdGeQpp, address:104.36.82.10:4101): protocol: rpc: dial tcp 104.36.82.10:4101: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:01:33.161690914 (58h25m42.884253954s) [seg_pos={0 0}, upload=56059]: Done waiting for segment. successful: 5 optimal: 14 errs: <nil> Jul 26 22:01:33.394119102 (58h25m43.116682142s) [upload=56059]: Done uploading segments... Jul 26 22:01:33.394147756 (58h25m43.116710806s) [upload=56059]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 5) Duration: 1m0.234194647s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [...................X....X......XX....XX.XX.XXX.X.XX.XXX.XXXX] duration: 30.231850229s FAIL (2025-07-26 22:01:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:00:31.091898029 (58h24m40.814461069s) [noise=true , address=77.90.31.11:4555 , piece_id=W226XY2ID57Z2RS2ZJYLRIAPYNRUEE5RL4ODO7WWEHHB3QCVR26Q, seg_pos={0 0}, upload=56051]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 188.166.225.170:54274->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:54274->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54274->77.90.31.11:4555: use of closed network connection Jul 26 22:00:31.091929348 (58h24m40.814492389s) [seg_pos={0 0}, upload=56051]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:00:31.319203675 (58h24m41.041766715s) [upload=56051]: Done uploading segments... Jul 26 22:00:31.319228783 (58h24m41.041791823s) [upload=56051]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 26 22:00:56.485759332 (58h25m6.208322372s) [upload=56069]: Starting upload Jul 26 22:00:56.485865435 (58h25m6.208428485s) [upload=56069]: Done starting upload Jul 26 22:00:56.485941100 (58h25m6.20850414s) [upload=56069]: Uploading segments... Jul 26 22:00:56.486112438 (58h25m6.208675478s) [upload=56069]: Got next segment. Inline: false Jul 26 22:00:56.486142405 (58h25m6.208705445s) [upload=56069]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:00:56.486147765 (58h25m6.208710825s) [upload=56069]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:00:56.720577592 (58h25m6.443140632s) [seg_pos={0 0}, upload=56069]: Begin upload segment... Jul 26 22:00:56.720618230 (58h25m6.44318127s) [seg_pos={0 0}, upload=56069]: Done begin upload segment. Jul 26 22:00:56.720639771 (58h25m6.443202811s) [noise=true , address=104.36.82.16:4295 , piece_id=F4BIRUOTBA4IV5WZFU7YUIYU6YO3G7I6STK5VDYL7XKX4BR5FCBQ, seg_pos={0 0}, upload=56069]: Uploading piece... Jul 26 22:00:56.720680018 (58h25m6.443243098s) [noise=true , address=45.91.116.8:4047 , piece_id=DLW6PBRQPIDBPS2LQSLLGEPKJQBX7O4V35ZE4754UIAOH5SGN7HA, seg_pos={0 0}, upload=56069]: Uploading piece... Jul 26 22:00:56.721007785 (58h25m6.443570825s) [noise=true , address=45.91.116.13:4155 , piece_id=JDBIHH3K32H4LPD2FME4OUIQ5OUQVBZ2F3JUPHGD5KIMIR3D4NAA, seg_pos={0 0}, upload=56069]: Uploading piece... Jul 26 22:00:56.721278062 (58h25m6.443841102s) [noise=true , address=45.91.116.12:4227 , piece_id=KUYTSKX2FK3JGPCTJBAPV5YYXGOTQT72TSD6NSTFBIREM5QNW76Q, seg_pos={0 0}, upload=56069]: Uploading piece... Jul 26 22:00:56.721548570 (58h25m6.44411161s) [noise=true , address=77.90.31.16:4733 , piece_id=XRTBWW6YJ3BNV7WWGQEUMQTOFF2B4DDG6XO74L3RGFEWGZIBL44Q, seg_pos={0 0}, upload=56069]: Uploading piece... Jul 26 22:00:56.721863162 (58h25m6.444426202s) [noise=true , address=45.91.116.7:4011 , piece_id=7WGV6TWHZUDX5ZOEZHLMJR2SOUYRQYGT3AJDZEBWULZXNCXIIIRA, seg_pos={0 0}, upload=56069]: Uploading piece... Jul 26 22:00:56.722085116 (58h25m6.444648156s) [noise=true , address=45.91.116.4:4939 , piece_id=J33L3H5ZWD2QZPRJQL4W77LOOTFT2MRT5OL3ZMC2QIY4MGJIKUWQ, seg_pos={0 0}, upload=56069]: Uploading piece... Jul 26 22:00:56.722297272 (58h25m6.444860312s) [noise=true , address=45.91.116.14:4190 , piece_id=PDDEVZ2HFE2ELUBOIJV4SNJ2GDQ4GJ32JBUGSYZI2EVDYP5L3GMQ, seg_pos={0 0}, upload=56069]: Uploading piece... Jul 26 22:00:56.722509859 (58h25m6.445072899s) [noise=true , address=77.90.31.10:4517 , piece_id=Z4D4M6FFI3PVITNTQGDQB6YMYNZEOX2GTD7SBRFKH3YGVMTLHMRA, seg_pos={0 0}, upload=56069]: Uploading piece... Jul 26 22:00:56.722734629 (58h25m6.445297669s) [noise=true , address=45.91.116.5:4299 , piece_id=MQC7P7XRFIQ5D5DAW72QIKNWL3E4ETY2V2MA3CM5NSGNW6VJB7GA, seg_pos={0 0}, upload=56069]: Uploading piece... Jul 26 22:00:56.722920494 (58h25m6.445483534s) [noise=true , address=45.91.116.12:4225 , piece_id=RUTB7JCY7VS5S4XRCQ75DK7IPICMGFO6ITUAY7B6ALQICUASA2UA, seg_pos={0 0}, upload=56069]: Uploading piece... Jul 26 22:00:56.723107432 (58h25m6.445670472s) [noise=true , address=104.36.82.9:4007 , piece_id=OW6JEUG67TKKQJ3YIW2UN6V22EZQ7B2G34UGSLMUA7VZMGTPQJIQ, seg_pos={0 0}, upload=56069]: Uploading piece... Jul 26 22:00:56.723354334 (58h25m6.445917364s) [noise=true , address=45.91.116.8:4048 , piece_id=5HTVLUNYQBWY4F33TU56BRI4L2WWI523BQD4JRTOHWFYJMAX3TPQ, seg_pos={0 0}, upload=56069]: Uploading piece... Jul 26 22:00:56.723557924 (58h25m6.446120954s) [noise=true , address=77.90.31.10:4518 , piece_id=2VZN35LTPMFXYF2B7ZK5G7OCAFHVTKGWPGAIBUJMVYB4SNGXFFXQ, seg_pos={0 0}, upload=56069]: Uploading piece... Jul 26 22:00:56.723750172 (58h25m6.446313212s) [noise=true , address=45.91.116.10:4081 , piece_id=ZG666UODQO5UONMNNWHHGLYW6O6HQQ3JNP5IANOPPH7NIC3XKBAQ, seg_pos={0 0}, upload=56069]: Uploading piece... Jul 26 22:00:56.723990772 (58h25m6.446553812s) [noise=true , address=104.36.82.9:4006 , piece_id=M74LPSZVRJYUMKPAJJHCTTJBU5A4OIJZ4HKSGHZ46EXNRJGOFFGA, seg_pos={0 0}, upload=56069]: Uploading piece... Jul 26 22:00:58.457433564 (58h25m8.179996604s) [upload=56069]: Next returned nil segment Jul 26 22:00:58.457450397 (58h25m8.180013437s) [upload=56069]: Waiting for error group managing segments... Jul 26 22:00:58.741054540 (58h25m8.46361762s) [noise=true , address=45.91.116.13:4155 , piece_id=JDBIHH3K32H4LPD2FME4OUIQ5OUQVBZ2F3JUPHGD5KIMIR3D4NAA, seg_pos={0 0}, upload=56069]: Done uploading piece. err: <nil> Jul 26 22:00:58.742847079 (58h25m8.46541011s) [noise=true , address=45.91.116.14:4190 , piece_id=PDDEVZ2HFE2ELUBOIJV4SNJ2GDQ4GJ32JBUGSYZI2EVDYP5L3GMQ, seg_pos={0 0}, upload=56069]: Done uploading piece. err: <nil> Jul 26 22:00:58.753442121 (58h25m8.476005151s) [noise=true , address=45.91.116.5:4299 , piece_id=MQC7P7XRFIQ5D5DAW72QIKNWL3E4ETY2V2MA3CM5NSGNW6VJB7GA, seg_pos={0 0}, upload=56069]: Done uploading piece. err: <nil> Jul 26 22:00:58.813674183 (58h25m8.536237313s) [noise=true , address=45.91.116.8:4047 , piece_id=DLW6PBRQPIDBPS2LQSLLGEPKJQBX7O4V35ZE4754UIAOH5SGN7HA, seg_pos={0 0}, upload=56069]: Done uploading piece. err: <nil> Jul 26 22:00:58.819903302 (58h25m8.542466342s) [noise=true , address=45.91.116.4:4939 , piece_id=J33L3H5ZWD2QZPRJQL4W77LOOTFT2MRT5OL3ZMC2QIY4MGJIKUWQ, seg_pos={0 0}, upload=56069]: Done uploading piece. err: <nil> Jul 26 22:00:58.844783495 (58h25m8.567346535s) [noise=true , address=45.91.116.10:4081 , piece_id=ZG666UODQO5UONMNNWHHGLYW6O6HQQ3JNP5IANOPPH7NIC3XKBAQ, seg_pos={0 0}, upload=56069]: Done uploading piece. err: <nil> Jul 26 22:00:58.846741832 (58h25m8.569304872s) [noise=true , address=45.91.116.8:4048 , piece_id=5HTVLUNYQBWY4F33TU56BRI4L2WWI523BQD4JRTOHWFYJMAX3TPQ, seg_pos={0 0}, upload=56069]: Done uploading piece. err: <nil> Jul 26 22:00:58.897336916 (58h25m8.619899946s) [noise=true , address=45.91.116.7:4011 , piece_id=7WGV6TWHZUDX5ZOEZHLMJR2SOUYRQYGT3AJDZEBWULZXNCXIIIRA, seg_pos={0 0}, upload=56069]: Done uploading piece. err: <nil> Jul 26 22:00:58.918214481 (58h25m8.640777521s) [noise=true , address=45.91.116.12:4227 , piece_id=KUYTSKX2FK3JGPCTJBAPV5YYXGOTQT72TSD6NSTFBIREM5QNW76Q, seg_pos={0 0}, upload=56069]: Done uploading piece. err: <nil> Jul 26 22:00:58.930555213 (58h25m8.653118253s) [noise=true , address=45.91.116.12:4225 , piece_id=RUTB7JCY7VS5S4XRCQ75DK7IPICMGFO6ITUAY7B6ALQICUASA2UA, seg_pos={0 0}, upload=56069]: Done uploading piece. err: <nil> Jul 26 22:01:08.140358367 (58h25m17.862921407s) [noise=true , address=77.90.31.10:4518 , piece_id=2VZN35LTPMFXYF2B7ZK5G7OCAFHVTKGWPGAIBUJMVYB4SNGXFFXQ, seg_pos={0 0}, upload=56069]: Done uploading piece. err: <nil> Jul 26 22:01:13.337945463 (58h25m23.060508504s) [noise=true , address=77.90.31.16:4733 , piece_id=XRTBWW6YJ3BNV7WWGQEUMQTOFF2B4DDG6XO74L3RGFEWGZIBL44Q, seg_pos={0 0}, upload=56069]: Done uploading piece. err: <nil> Jul 26 22:01:25.842355036 (58h25m35.564918076s) [noise=true , address=77.90.31.10:4517 , piece_id=Z4D4M6FFI3PVITNTQGDQB6YMYNZEOX2GTD7SBRFKH3YGVMTLHMRA, seg_pos={0 0}, upload=56069]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:01:25.842777624 (58h25m35.565340674s) [noise=true , address=104.36.82.9:4007 , piece_id=OW6JEUG67TKKQJ3YIW2UN6V22EZQ7B2G34UGSLMUA7VZMGTPQJIQ, seg_pos={0 0}, upload=56069]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:01:25.842816089 (58h25m35.565379129s) [noise=true , address=104.36.82.9:4006 , piece_id=M74LPSZVRJYUMKPAJJHCTTJBU5A4OIJZ4HKSGHZ46EXNRJGOFFGA, seg_pos={0 0}, upload=56069]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:01:25.843162521 (58h25m35.565725561s) [noise=true , address=104.36.82.16:4295 , piece_id=F4BIRUOTBA4IV5WZFU7YUIYU6YO3G7I6STK5VDYL7XKX4BR5FCBQ, seg_pos={0 0}, upload=56069]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:55504->104.36.82.16:4295: use of closed network connection; conn closed: op Jul 26 22:01:25.843191757 (58h25m35.565754797s) [seg_pos={0 0}, upload=56069]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:01:26.072871977 (58h25m35.795435027s) [upload=56069]: Done uploading segments... Jul 26 22:01:26.072902745 (58h25m35.795465785s) [upload=56069]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.231872631s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..X.................X.......................................] duration: 1.8866692s FAIL (2025-07-26 22:01:21 UTC) orbiter warning: digest mismatch: after 0783c6a051cf3b535b7c61f8e6f0cfe751970c7121d9bce2182fd51e29ca8fc6 != before ee59801cb45aa958945ed73fcee20d9c3604e7f558c75f49df119aca36a29d88 Duration: 1.886697816s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 19 out of 60 is failed (threshold=3) [..........................XX.........XX.XX.XXXX.XX.XXXXX.X.X] duration: 30.244392732s FAIL (2025-07-26 22:01:13 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.244414263s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X..X......] duration: 22.33322425s FAIL (2025-07-26 22:01:00 UTC) orbiter warning: context deadline exceeded Duration: 22.333247243s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...X.................X......................................] duration: 1.858746328s FAIL (2025-07-26 22:00:38 UTC) orbiter warning: digest mismatch: after 0783c6a051cf3b535b7c61f8e6f0cfe751970c7121d9bce2182fd51e29ca8fc6 != before ee59801cb45aa958945ed73fcee20d9c3604e7f558c75f49df119aca36a29d88 Duration: 1.858763742s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X..X.....] duration: 13.899972095s FAIL (2025-07-26 22:00:32 UTC) orbiter warning: context deadline exceeded Duration: 13.899990832s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [....................X....X......XX....XX.XX.XXX.X.XX.XXX.XXX] duration: 30.229540183s FAIL (2025-07-26 22:00:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== id=DISWA4JHXGUS7TUPFYUJUQ2KK4W4CCUB4K62UNHJRNQCPFJZSGRQ, seg_pos={0 0}, upload=56042]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:00:00.620197447 (58h24m10.342760487s) [seg_pos={0 0}, upload=56042]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 22:00:00.852187346 (58h24m10.574750386s) [upload=56042]: Done uploading segments... Jul 26 22:00:00.852213657 (58h24m10.574776697s) [upload=56042]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 26 22:00:01.733345636 (58h24m11.455908686s) [upload=56051]: Starting upload Jul 26 22:00:01.733421340 (58h24m11.45598439s) [upload=56051]: Done starting upload Jul 26 22:00:01.733520490 (58h24m11.45608354s) [upload=56051]: Uploading segments... Jul 26 22:00:01.733677741 (58h24m11.456240801s) [upload=56051]: Got next segment. Inline: false Jul 26 22:00:01.733734750 (58h24m11.4562978s) [upload=56051]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:00:01.733742224 (58h24m11.456305274s) [upload=56051]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:00:01.972940214 (58h24m11.695503274s) [seg_pos={0 0}, upload=56051]: Begin upload segment... Jul 26 22:00:01.973011690 (58h24m11.69557474s) [seg_pos={0 0}, upload=56051]: Done begin upload segment. Jul 26 22:00:01.973035837 (58h24m11.695598897s) [noise=true , address=104.36.82.4:4801 , piece_id=2FCQQHRB5YFZ3LNRMNVJ222S7PM3EMYG7PM7F5NGFXNOQ6OBE3IA, seg_pos={0 0}, upload=56051]: Uploading piece... Jul 26 22:00:01.973073399 (58h24m11.695636449s) [noise=true , address=45.91.116.13:4153 , piece_id=PKVG2H5BN7NPADLZ66FIDJO2IP5ZRY7M2F4HAOJVRSNANJW2VWKA, seg_pos={0 0}, upload=56051]: Uploading piece... Jul 26 22:00:01.973666564 (58h24m11.696229614s) [noise=true , address=45.91.116.12:4227 , piece_id=VKNJZ26CDHY7EHG3NDGCVPXV3R66KYBMWVP3XJXSDNBHPOA3SKCQ, seg_pos={0 0}, upload=56051]: Uploading piece... Jul 26 22:00:01.973684368 (58h24m11.696247418s) [noise=true , address=45.91.116.8:4045 , piece_id=MGOS7DVTH6WR52AZS5X45VCG3ELPGS54QBOUC5ZUISYTS2PRFIGA, seg_pos={0 0}, upload=56051]: Uploading piece... Jul 26 22:00:01.974133798 (58h24m11.696696848s) [noise=true , address=45.91.116.7:4011 , piece_id=AOK5TGTDFCZ2W6AS2CYURJOKPAGZ2GVNUYUXAZQASAFPDJ76SI4A, seg_pos={0 0}, upload=56051]: Uploading piece... Jul 26 22:00:01.974843836 (58h24m11.697406886s) [noise=true , address=77.90.31.11:4555 , piece_id=W226XY2ID57Z2RS2ZJYLRIAPYNRUEE5RL4ODO7WWEHHB3QCVR26Q, seg_pos={0 0}, upload=56051]: Uploading piece... Jul 26 22:00:01.975095367 (58h24m11.697658427s) [noise=true , address=77.90.31.16:4735 , piece_id=66D4LKOHOP62OH2F7M5NNPFR2YI5UVSADMZKOISXF6J4FUK3B3GQ, seg_pos={0 0}, upload=56051]: Uploading piece... Jul 26 22:00:01.975447181 (58h24m11.698010231s) [noise=true , address=45.91.116.6:4973 , piece_id=C6AIVOYBCBCVFMDSY5VRVUE3DKBY7LHR6AGM53KDLDWWQ7VLT3PQ, seg_pos={0 0}, upload=56051]: Uploading piece... Jul 26 22:00:01.975640170 (58h24m11.69820322s) [noise=true , address=45.91.116.4:4938 , piece_id=B7IW2K3TBLOSKUMUR3ZMMB2JPE6KPLYVT6VQN6YURKZIPE5ASTFA, seg_pos={0 0}, upload=56051]: Uploading piece... Jul 26 22:00:01.976052188 (58h24m11.698615268s) [noise=true , address=45.91.116.9:4262 , piece_id=TSLSIFVGETK3TKZYLJMMKO6NRYFZS4S4AA4Q5IOMPP73Q7K44RSA, seg_pos={0 0}, upload=56051]: Uploading piece... Jul 26 22:00:01.976101533 (58h24m11.698664583s) [noise=true , address=77.90.31.12:4590 , piece_id=H5RPBTCDBRQXLBKIQIMLLKV5KUCYWG77NQE7HADGMNIC3FYQHKEQ, seg_pos={0 0}, upload=56051]: Uploading piece... Jul 26 22:00:01.976728943 (58h24m11.699291984s) [noise=true , address=104.36.82.8:4972 , piece_id=QIZAWD7CF27NQ3DHJVKIFMOL7VQTFI54T4JWRNJXNDU67AISRZ2Q, seg_pos={0 0}, upload=56051]: Uploading piece... Jul 26 22:00:01.977229812 (58h24m11.699792862s) [noise=true , address=45.91.116.13:4155 , piece_id=VLOMIWXYVO23SO3T37ZK5GGPLLEZWPWZOFBYZQCJXCN2KCFMZE2Q, seg_pos={0 0}, upload=56051]: Uploading piece... Jul 26 22:00:01.978034821 (58h24m11.700597872s) [noise=true , address=77.90.31.12:4589 , piece_id=FQ4O6TTNAK6EKKTR2FEZXUVNUKPB2ZIJ2X4LD5MJGHEVUBP75XOA, seg_pos={0 0}, upload=56051]: Uploading piece... Jul 26 22:00:01.978516564 (58h24m11.701079614s) [noise=true , address=77.90.31.15:4697 , piece_id=WN4BUJJITUKNCDEBSOZSXT35PKS52BI2Y7NQNTICZBFYLKTQZFHA, seg_pos={0 0}, upload=56051]: Uploading piece... Jul 26 22:00:01.979028973 (58h24m11.701592024s) [noise=true , address=104.36.82.14:4223 , piece_id=E7Y4JCIRX773H2X2IDBTVQ6M2WM4KK5METNAHYTTR6JXYEPAY6GQ, seg_pos={0 0}, upload=56051]: Uploading piece... Jul 26 22:00:03.603531782 (58h24m13.326094832s) [upload=56051]: Next returned nil segment Jul 26 22:00:03.603581766 (58h24m13.326144817s) [upload=56051]: Waiting for error group managing segments... Jul 26 22:00:03.992813067 (58h24m13.715376117s) [noise=true , address=45.91.116.12:4227 , piece_id=VKNJZ26CDHY7EHG3NDGCVPXV3R66KYBMWVP3XJXSDNBHPOA3SKCQ, seg_pos={0 0}, upload=56051]: Done uploading piece. err: <nil> Jul 26 22:00:03.998463407 (58h24m13.721026467s) [noise=true , address=45.91.116.4:4938 , piece_id=B7IW2K3TBLOSKUMUR3ZMMB2JPE6KPLYVT6VQN6YURKZIPE5ASTFA, seg_pos={0 0}, upload=56051]: Done uploading piece. err: <nil> Jul 26 22:00:03.998640316 (58h24m13.721203356s) [noise=true , address=45.91.116.7:4011 , piece_id=AOK5TGTDFCZ2W6AS2CYURJOKPAGZ2GVNUYUXAZQASAFPDJ76SI4A, seg_pos={0 0}, upload=56051]: Done uploading piece. err: <nil> Jul 26 22:00:04.069056994 (58h24m13.791620045s) [noise=true , address=45.91.116.13:4153 , piece_id=PKVG2H5BN7NPADLZ66FIDJO2IP5ZRY7M2F4HAOJVRSNANJW2VWKA, seg_pos={0 0}, upload=56051]: Done uploading piece. err: <nil> Jul 26 22:00:04.184278089 (58h24m13.906841139s) [noise=true , address=45.91.116.8:4045 , piece_id=MGOS7DVTH6WR52AZS5X45VCG3ELPGS54QBOUC5ZUISYTS2PRFIGA, seg_pos={0 0}, upload=56051]: Done uploading piece. err: <nil> Jul 26 22:00:04.208102603 (58h24m13.930665643s) [noise=true , address=45.91.116.9:4262 , piece_id=TSLSIFVGETK3TKZYLJMMKO6NRYFZS4S4AA4Q5IOMPP73Q7K44RSA, seg_pos={0 0}, upload=56051]: Done uploading piece. err: <nil> Jul 26 22:00:04.216424424 (58h24m13.938987464s) [noise=true , address=45.91.116.6:4973 , piece_id=C6AIVOYBCBCVFMDSY5VRVUE3DKBY7LHR6AGM53KDLDWWQ7VLT3PQ, seg_pos={0 0}, upload=56051]: Done uploading piece. err: <nil> Jul 26 22:00:04.310594170 (58h24m14.03315722s) [noise=true , address=45.91.116.13:4155 , piece_id=VLOMIWXYVO23SO3T37ZK5GGPLLEZWPWZOFBYZQCJXCN2KCFMZE2Q, seg_pos={0 0}, upload=56051]: Done uploading piece. err: <nil> Jul 26 22:00:13.044830285 (58h24m22.767393315s) [noise=true , address=77.90.31.15:4697 , piece_id=WN4BUJJITUKNCDEBSOZSXT35PKS52BI2Y7NQNTICZBFYLKTQZFHA, seg_pos={0 0}, upload=56051]: Done uploading piece. err: <nil> Jul 26 22:00:13.412058697 (58h24m23.134621737s) [noise=true , address=77.90.31.12:4589 , piece_id=FQ4O6TTNAK6EKKTR2FEZXUVNUKPB2ZIJ2X4LD5MJGHEVUBP75XOA, seg_pos={0 0}, upload=56051]: Done uploading piece. err: <nil> Jul 26 22:00:20.964769299 (58h24m30.687332389s) [noise=true , address=77.90.31.16:4735 , piece_id=66D4LKOHOP62OH2F7M5NNPFR2YI5UVSADMZKOISXF6J4FUK3B3GQ, seg_pos={0 0}, upload=56051]: Done uploading piece. err: <nil> Jul 26 22:00:29.835240626 (58h24m39.557803666s) [noise=true , address=77.90.31.12:4590 , piece_id=H5RPBTCDBRQXLBKIQIMLLKV5KUCYWG77NQE7HADGMNIC3FYQHKEQ, seg_pos={0 0}, upload=56051]: Done uploading piece. err: <nil> Jul 26 22:00:31.091772558 (58h24m40.814335598s) [noise=true , address=104.36.82.4:4801 , piece_id=2FCQQHRB5YFZ3LNRMNVJ222S7PM3EMYG7PM7F5NGFXNOQ6OBE3IA, seg_pos={0 0}, upload=56051]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 188.166.225.170:55656->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:00:31.091862340 (58h24m40.81442537s) [noise=true , address=104.36.82.14:4223 , piece_id=E7Y4JCIRX773H2X2IDBTVQ6M2WM4KK5METNAHYTTR6JXYEPAY6GQ, seg_pos={0 0}, upload=56051]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:44160->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:00:31.091867450 (58h24m40.81443049s) [noise=true , address=104.36.82.8:4972 , piece_id=QIZAWD7CF27NQ3DHJVKIFMOL7VQTFI54T4JWRNJXNDU67AISRZ2Q, seg_pos={0 0}, upload=56051]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 188.166.225.170:51058->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:00:31.091898029 (58h24m40.814461069s) [noise=true , address=77.90.31.11:4555 , piece_id=W226XY2ID57Z2RS2ZJYLRIAPYNRUEE5RL4ODO7WWEHHB3QCVR26Q, seg_pos={0 0}, upload=56051]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: write tcp 188.166.225.170:54274->77.90.31.11:4555: use of closed network connection; write tcp 188.166.225.170:54274->77.90.31.11:4555: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54274->77.90.31.11:4555: use of closed network connection Jul 26 22:00:31.091929348 (58h24m40.814492389s) [seg_pos={0 0}, upload=56051]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:00:31.319203675 (58h24m41.041766715s) [upload=56051]: Done uploading segments... Jul 26 22:00:31.319228783 (58h24m41.041791823s) [upload=56051]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.22955449s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 18 out of 60 is failed (threshold=3) [...........................XX.........XX.XX.XXXX.XX.XXXXX.X.] duration: 22.489626233s FAIL (2025-07-26 22:00:31 UTC) orbiter warning: uplink: context deadline exceeded Duration: 22.489659887s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 31 out of 60 is failed (threshold=3) [.............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.230535266s FAIL (2025-07-26 22:00:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== FWRREFJZTOL4I2RXQWPKHW6BXBA, seg_pos={0 0}, upload=56040]: Uploading piece... Jul 26 21:59:25.612013916 (58h23m35.334576956s) [noise=true , address=104.36.82.51:4072 , piece_id=KRKWQFVAN7WYSD2RNUYYBXVQICK5IVCEMIPRPR4JNBMXJC6YN5XQ, seg_pos={0 0}, upload=56040]: Uploading piece... Jul 26 21:59:25.612280135 (58h23m35.334843175s) [noise=true , address=79.127.205.246:4278 , piece_id=RKDBAO2WN4NUYZNKMJ3BERPUWJQRNFVMAUHEKQJZGLGRQB2IEAVA, seg_pos={0 0}, upload=56040]: Uploading piece... Jul 26 21:59:25.612450711 (58h23m35.335013751s) [noise=true , address=64.49.13.138:4483 , piece_id=YSQHL5CDNPXMPZ2BA66XPA6WKIXFP7UUJ3ESLBSL2YLFWKZFYYLQ, seg_pos={0 0}, upload=56040]: Uploading piece... Jul 26 21:59:25.612585690 (58h23m35.33514873s) [noise=true , address=77.90.31.11:4558 , piece_id=RHJZ6TCRCEYQXYO7ILC2CRPYART4WVJW6QIT5K7DO6QKLYOFFBJQ, seg_pos={0 0}, upload=56040]: Uploading piece... Jul 26 21:59:25.612695710 (58h23m35.33525876s) [noise=true , address=104.36.82.53:4138 , piece_id=HTDKTVBFWBQ4W7MGLKDGVNNT2ZTNLNZ6RZZPEFRU2CC6GZXF5M4Q, seg_pos={0 0}, upload=56040]: Uploading piece... Jul 26 21:59:25.612868892 (58h23m35.335431922s) [noise=true , address=104.36.82.52:4099 , piece_id=X3ODAXQ765FA4OOPZE2ODBESHXT6W3NBR3Q5BSMSUFOTYABQTJFA, seg_pos={0 0}, upload=56040]: Uploading piece... Jul 26 21:59:25.613003469 (58h23m35.335566509s) [noise=true , address=79.127.201.215:4354 , piece_id=BYVKTMEFZENOY7DSNB4UOXX7FCWCXP3KCBH2KWIZXUH7JA7HPUNQ, seg_pos={0 0}, upload=56040]: Uploading piece... Jul 26 21:59:25.613339793 (58h23m35.335902833s) [noise=true , address=104.36.82.22:4524 , piece_id=CUBTAP2FXFWGVAZ3RE7G544NRR74VQVQVSGCS6YOI33EFAXHKZNQ, seg_pos={0 0}, upload=56040]: Uploading piece... Jul 26 21:59:25.613577628 (58h23m35.336140668s) [noise=true , address=104.36.82.54:4160 , piece_id=QWUOVCCILW2I6UTBKJQ4HKJ546Z3FULYU2PDUIE5G5I6PETBEBRQ, seg_pos={0 0}, upload=56040]: Uploading piece... Jul 26 21:59:25.613613497 (58h23m35.336176537s) [noise=true , address=104.36.82.15:4280 , piece_id=U7S2MYZLWEGXPK624STXFUMNEB3F3NOWFXK3XUCGUZSQD3POSSLA, seg_pos={0 0}, upload=56040]: Uploading piece... Jul 26 21:59:25.613785375 (58h23m35.336348416s) [noise=true , address=77.90.31.21:4908 , piece_id=7MRZLNXQTYO6CUK3ZDF6QIVYBNDST4RX4ZJWHDJVLK5NJSJVAQZQ, seg_pos={0 0}, upload=56040]: Uploading piece... Jul 26 21:59:25.614028150 (58h23m35.33659121s) [noise=true , address=79.127.219.51:4201 , piece_id=I7XACGD55BQEKZ7WWFPNPKVR5LYP72NBPR3R6W4IECPRQQKX2CVQ, seg_pos={0 0}, upload=56040]: Uploading piece... Jul 26 21:59:25.614235065 (58h23m35.336798106s) [noise=true , address=104.36.82.47:4546 , piece_id=XEIYDJHYTKXM5P4A33HP6E5GMFDKR5COKOS374T5D4OTFJML3QOA, seg_pos={0 0}, upload=56040]: Uploading piece... Jul 26 21:59:25.614269541 (58h23m35.336832571s) [noise=true , address=104.36.82.46:4483 , piece_id=2GFCH32ZX2OONXZJ2AULAT6HRE45WGTBJJAGALC7UPN7YEXLELJA, seg_pos={0 0}, upload=56040]: Uploading piece... Jul 26 21:59:26.712216394 (58h23m36.434779434s) [upload=56040]: Next returned nil segment Jul 26 21:59:26.712234197 (58h23m36.434797237s) [upload=56040]: Waiting for error group managing segments... Jul 26 21:59:27.293574774 (58h23m37.016137824s) [noise=true , address=79.127.205.246:4278 , piece_id=RKDBAO2WN4NUYZNKMJ3BERPUWJQRNFVMAUHEKQJZGLGRQB2IEAVA, seg_pos={0 0}, upload=56040]: Done uploading piece. err: <nil> Jul 26 21:59:27.472232291 (58h23m37.194795321s) [noise=true , address=79.127.205.234:4613 , piece_id=KHCNIFSPE46USNR6DCNNPX4CFBYP23RF7BXCRVNAJ76XN4Y2U6PA, seg_pos={0 0}, upload=56040]: Done uploading piece. err: <nil> Jul 26 21:59:30.936113020 (58h23m40.65867606s) [noise=true , address=79.127.219.51:4201 , piece_id=I7XACGD55BQEKZ7WWFPNPKVR5LYP72NBPR3R6W4IECPRQQKX2CVQ, seg_pos={0 0}, upload=56040]: Done uploading piece. err: <nil> Jul 26 21:59:31.364105923 (58h23m41.086668963s) [noise=true , address=64.49.13.138:4483 , piece_id=YSQHL5CDNPXMPZ2BA66XPA6WKIXFP7UUJ3ESLBSL2YLFWKZFYYLQ, seg_pos={0 0}, upload=56040]: Done uploading piece. err: <nil> Jul 26 21:59:34.416742093 (58h23m44.139305134s) [noise=true , address=79.127.201.215:4354 , piece_id=BYVKTMEFZENOY7DSNB4UOXX7FCWCXP3KCBH2KWIZXUH7JA7HPUNQ, seg_pos={0 0}, upload=56040]: Done uploading piece. err: <nil> Jul 26 21:59:34.906729680 (58h23m44.629292711s) [noise=true , address=156.146.43.225:4449 , piece_id=MTF472LBY2TDOWLN6CBVB6OUAFWRREFJZTOL4I2RXQWPKHW6BXBA, seg_pos={0 0}, upload=56040]: Done uploading piece. err: <nil> Jul 26 21:59:42.804807210 (58h23m52.52737025s) [noise=true , address=77.90.31.11:4558 , piece_id=RHJZ6TCRCEYQXYO7ILC2CRPYART4WVJW6QIT5K7DO6QKLYOFFBJQ, seg_pos={0 0}, upload=56040]: Done uploading piece. err: <nil> Jul 26 22:00:24.519959779 (58h24m34.242522819s) [noise=true , address=104.36.82.15:4280 , piece_id=U7S2MYZLWEGXPK624STXFUMNEB3F3NOWFXK3XUCGUZSQD3POSSLA, seg_pos={0 0}, upload=56040]: Done uploading piece. err: ecclient: upload failed (node:1t4hMpMGCu1YMmNLX6fssTjWy9Xi5Fp8kdHTDmkhxH9qsZiYM6, address:104.36.82.15:4280): protocol: rpc: dial tcp 104.36.82.15:4280: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:00:24.520014243 (58h24m34.242577283s) [noise=true , address=104.36.82.56:4245 , piece_id=VOVUBOH6UHL4XTBEVBKMGI56TJ2HPKVGGSE6BG4T2OMFTT7D6T7A, seg_pos={0 0}, upload=56040]: Done uploading piece. err: ecclient: upload failed (node:12YnxyH9TZVuVheQpH9NVRfFUeWHJEtV97wyEJafhCzGZSv13Fn, address:104.36.82.56:4245): protocol: rpc: dial tcp 104.36.82.56:4245: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:00:24.520045774 (58h24m34.242608815s) [noise=true , address=104.36.82.35:4601 , piece_id=PFURG7AQOZCKMRTH5BKGW6BKHKU4YWN75AXBLBUQAFQTYQYLNRVA, seg_pos={0 0}, upload=56040]: Done uploading piece. err: ecclient: upload failed (node:16ggANPGWDLtyxp9hMkCsNYueB2KuVWsKVrG8mFa6pPBXW5RiW, address:104.36.82.35:4601): protocol: rpc: dial tcp 104.36.82.35:4601: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:00:24.520149232 (58h24m34.242712272s) [noise=true , address=104.36.82.46:4483 , piece_id=2GFCH32ZX2OONXZJ2AULAT6HRE45WGTBJJAGALC7UPN7YEXLELJA, seg_pos={0 0}, upload=56040]: Done uploading piece. err: ecclient: upload failed (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY, address:104.36.82.46:4483): protocol: rpc: dial tcp 104.36.82.46:4483: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:00:24.520181092 (58h24m34.242744143s) [noise=true , address=104.36.82.53:4138 , piece_id=HTDKTVBFWBQ4W7MGLKDGVNNT2ZTNLNZ6RZZPEFRU2CC6GZXF5M4Q, seg_pos={0 0}, upload=56040]: Done uploading piece. err: ecclient: upload failed (node:1YUGGJg6mWeQMmzSNWtLFWAaWwaKWfEf8wxXs1jwiZaA9nhsr3, address:104.36.82.53:4138): protocol: rpc: dial tcp 104.36.82.53:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:00:24.520211301 (58h24m34.242774341s) [noise=true , address=104.36.82.52:4099 , piece_id=X3ODAXQ765FA4OOPZE2ODBESHXT6W3NBR3Q5BSMSUFOTYABQTJFA, seg_pos={0 0}, upload=56040]: Done uploading piece. err: ecclient: upload failed (node:122dGFWNmsuheRmfEfNGWHftPpCcWv7HHXKymgJxsye6P7ZKhZT, address:104.36.82.52:4099): protocol: rpc: dial tcp 104.36.82.52:4099: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:00:24.520249063 (58h24m34.242812103s) [noise=true , address=104.36.82.54:4160 , piece_id=QWUOVCCILW2I6UTBKJQ4HKJ546Z3FULYU2PDUIE5G5I6PETBEBRQ, seg_pos={0 0}, upload=56040]: Done uploading piece. err: ecclient: upload failed (node:12qRsRx7vmb3S9m5J7gceK44gnuuYYMyorSAyxcpjpPeRJJFZ84, address:104.36.82.54:4160): protocol: rpc: dial tcp 104.36.82.54:4160: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:00:24.520279241 (58h24m34.242842281s) [noise=true , address=104.36.82.47:4546 , piece_id=XEIYDJHYTKXM5P4A33HP6E5GMFDKR5COKOS374T5D4OTFJML3QOA, seg_pos={0 0}, upload=56040]: Done uploading piece. err: ecclient: upload failed (node:1FJGmqUskGmxYctVCEaYhRHq98m1VCVZiWL1ShEMJzmieaYo1u, address:104.36.82.47:4546): protocol: rpc: dial tcp 104.36.82.47:4546: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:00:24.520363612 (58h24m34.242926652s) [noise=true , address=104.36.82.51:4072 , piece_id=KRKWQFVAN7WYSD2RNUYYBXVQICK5IVCEMIPRPR4JNBMXJC6YN5XQ, seg_pos={0 0}, upload=56040]: Done uploading piece. err: ecclient: upload failed (node:12FSQUJNBPY52p2Wkg48DWPN3bdciabNacXcgDEHSwLRrnPG9ne, address:104.36.82.51:4072): protocol: rpc: dial tcp 104.36.82.51:4072: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:00:24.520403429 (58h24m34.242966469s) [noise=true , address=104.36.82.22:4524 , piece_id=CUBTAP2FXFWGVAZ3RE7G544NRR74VQVQVSGCS6YOI33EFAXHKZNQ, seg_pos={0 0}, upload=56040]: Done uploading piece. err: ecclient: upload failed (node:1Kb5FifYvHa4UijPvLu4cNbH8nXznoi1gyBYnPd2s6nizaGJuk, address:104.36.82.22:4524): protocol: write tcp 188.166.225.170:45176->104.36.82.22:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:00:24.520437353 (58h24m34.243000393s) [noise=true , address=77.90.31.21:4908 , piece_id=7MRZLNXQTYO6CUK3ZDF6QIVYBNDST4RX4ZJWHDJVLK5NJSJVAQZQ, seg_pos={0 0}, upload=56040]: Done uploading piece. err: ecclient: upload failed (node:13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga, address:77.90.31.21:4908): protocol: read tcp 188.166.225.170:54432->77.90.31.21:4908: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:00:24.520454075 (58h24m34.243017115s) [seg_pos={0 0}, upload=56040]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 26 22:00:24.747950406 (58h24m34.470513446s) [upload=56040]: Done uploading segments... Jul 26 22:00:24.747976897 (58h24m34.470539937s) [upload=56040]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.230548181s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.....................X....X......XX....XX.XX.XXX.X.XX.XXX.XX] duration: 30.234630452s FAIL (2025-07-26 22:00:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== oise=true , address=104.36.82.8:4970 , piece_id=HTN6FFOYG5KVMLGN5SKOCQGSOZAMGBKFPRVYB6XCSJIKHSCCBTMA, seg_pos={0 0}, upload=56022]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:58:46.879499539 (58h22m56.602062579s) [seg_pos={0 0}, upload=56022]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 26 21:58:47.106507134 (58h22m56.829070174s) [upload=56022]: Done uploading segments... Jul 26 21:58:47.106530378 (58h22m56.829093418s) [upload=56022]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Jul 26 21:59:31.268112318 (58h23m40.990675358s) [upload=56042]: Starting upload Jul 26 21:59:31.268171280 (58h23m40.99073431s) [upload=56042]: Done starting upload Jul 26 21:59:31.268332619 (58h23m40.990895679s) [upload=56042]: Uploading segments... Jul 26 21:59:31.268456115 (58h23m40.991019165s) [upload=56042]: Got next segment. Inline: false Jul 26 21:59:31.268517222 (58h23m40.991080272s) [upload=56042]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 21:59:31.268523575 (58h23m40.991086615s) [upload=56042]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 21:59:31.502282497 (58h23m41.224845537s) [seg_pos={0 0}, upload=56042]: Begin upload segment... Jul 26 21:59:31.502347081 (58h23m41.224910121s) [seg_pos={0 0}, upload=56042]: Done begin upload segment. Jul 26 21:59:31.502370465 (58h23m41.224933505s) [noise=true , address=45.91.116.5:4298 , piece_id=GOQGN6CIY5QR5XTDH2C3BD5EKKIL7XNOVIEUUIBBJ6GUXJEF5WKQ, seg_pos={0 0}, upload=56042]: Uploading piece... Jul 26 21:59:31.502408959 (58h23m41.224971989s) [noise=true , address=45.91.116.5:4297 , piece_id=542GSYWXEKL57Z35LWY2G3FKNBHSIAOX773CSR5EBRZJGBQEVYQQ, seg_pos={0 0}, upload=56042]: Uploading piece... Jul 26 21:59:31.502721818 (58h23m41.225284858s) [noise=true , address=45.91.116.8:4048 , piece_id=OAJ4H7OOUQU6L5T2PBY7B375ZOPRHA425C2YI5SK7HEE4VXLVR4Q, seg_pos={0 0}, upload=56042]: Uploading piece... Jul 26 21:59:31.502881964 (58h23m41.225445004s) [noise=true , address=45.91.116.12:4226 , piece_id=MGDXGUZ3YMWANOW2O77VWXBXENUPJTJZMCDBQ4D3GPQNO7STAEJQ, seg_pos={0 0}, upload=56042]: Uploading piece... Jul 26 21:59:31.502977908 (58h23m41.225540948s) [noise=true , address=104.36.82.2:4730 , piece_id=ETEMO7KOSUDPD37XNKYFOXOKV7PHZIFAVU4X7KLF7WNZOU56PFAQ, seg_pos={0 0}, upload=56042]: Uploading piece... Jul 26 21:59:31.503220742 (58h23m41.225783782s) [noise=true , address=45.91.116.13:4154 , piece_id=2RJKH27HHTUL32USKWGHY5CRXIMYJTEIERZ3DL2TBSHS4I2COWNQ, seg_pos={0 0}, upload=56042]: Uploading piece... Jul 26 21:59:31.503471863 (58h23m41.226034903s) [noise=true , address=77.90.31.19:4842 , piece_id=CVXEA2ET3W62HQS6652SZZNPBXWD3E5HJH6VV7HQXL5O2VUCC4FA, seg_pos={0 0}, upload=56042]: Uploading piece... Jul 26 21:59:31.503786144 (58h23m41.226349174s) [noise=true , address=45.91.116.8:4045 , piece_id=IN7CT5LZFPRGGQLX2ZYYLQGXV57CUDEG3VOAPRIL5BFKIHJU4FRQ, seg_pos={0 0}, upload=56042]: Uploading piece... Jul 26 21:59:31.503942223 (58h23m41.226505263s) [noise=true , address=45.91.116.6:4973 , piece_id=KBXWDUZO3GRJFI5O6K4SSQPOQBTFV4CXQJSDYC6QEM5AMY5H5SMA, seg_pos={0 0}, upload=56042]: Uploading piece... Jul 26 21:59:31.504183163 (58h23m41.226746193s) [noise=true , address=45.91.116.10:4081 , piece_id=PCIFS5JSU5UJGUBAAEUU5OWTTQK4D4OPWSUEYARXIRNVANONIR2Q, seg_pos={0 0}, upload=56042]: Uploading piece... Jul 26 21:59:31.504462920 (58h23m41.22702596s) [noise=true , address=45.91.116.13:4153 , piece_id=2X4PGTRSD3OSOKORNDCRL4C7FP5IEBX5LWLYMHLZSFCYPR2L5PAA, seg_pos={0 0}, upload=56042]: Uploading piece... Jul 26 21:59:31.504602857 (58h23m41.227165897s) [noise=true , address=45.91.116.9:4263 , piece_id=OO4PBROVPCW2KVJKE6GYKI4MZT2VGXLTTLV65RXILYZRLQH6HRVA, seg_pos={0 0}, upload=56042]: Uploading piece... Jul 26 21:59:31.504834009 (58h23m41.227397049s) [noise=true , address=104.36.82.4:4802 , piece_id=DISWA4JHXGUS7TUPFYUJUQ2KK4W4CCUB4K62UNHJRNQCPFJZSGRQ, seg_pos={0 0}, upload=56042]: Uploading piece... Jul 26 21:59:31.505042558 (58h23m41.227605599s) [noise=true , address=104.36.82.16:4295 , piece_id=3QGMQQ4QPG6S5DB6I26BCVCX3NL4Q2W6VRCPH34SRC7XO6NZ6EHA, seg_pos={0 0}, upload=56042]: Uploading piece... Jul 26 21:59:31.505081583 (58h23m41.227644623s) [noise=true , address=45.91.116.14:4190 , piece_id=YPCKIIS6LTUTIB6CMJ2ZWKPSUV5T5VUEXNUYQKUP77OALAUP43UQ, seg_pos={0 0}, upload=56042]: Uploading piece... Jul 26 21:59:31.505319158 (58h23m41.227882198s) [noise=true , address=104.36.82.9:4007 , piece_id=TN35FQD6U5HWE6DEIPW2UGTCY4EVPA2F7GHLY5KQ5NTKO7Y3F5XA, seg_pos={0 0}, upload=56042]: Uploading piece... Jul 26 21:59:33.224153150 (58h23m42.94671619s) [upload=56042]: Next returned nil segment Jul 26 21:59:33.224173659 (58h23m42.94673672s) [upload=56042]: Waiting for error group managing segments... Jul 26 21:59:33.518832864 (58h23m43.241395904s) [noise=true , address=45.91.116.14:4190 , piece_id=YPCKIIS6LTUTIB6CMJ2ZWKPSUV5T5VUEXNUYQKUP77OALAUP43UQ, seg_pos={0 0}, upload=56042]: Done uploading piece. err: <nil> Jul 26 21:59:33.519405469 (58h23m43.241968509s) [noise=true , address=45.91.116.9:4263 , piece_id=OO4PBROVPCW2KVJKE6GYKI4MZT2VGXLTTLV65RXILYZRLQH6HRVA, seg_pos={0 0}, upload=56042]: Done uploading piece. err: <nil> Jul 26 21:59:33.520382628 (58h23m43.242945668s) [noise=true , address=45.91.116.5:4298 , piece_id=GOQGN6CIY5QR5XTDH2C3BD5EKKIL7XNOVIEUUIBBJ6GUXJEF5WKQ, seg_pos={0 0}, upload=56042]: Done uploading piece. err: <nil> Jul 26 21:59:33.536388323 (58h23m43.258951363s) [noise=true , address=45.91.116.8:4045 , piece_id=IN7CT5LZFPRGGQLX2ZYYLQGXV57CUDEG3VOAPRIL5BFKIHJU4FRQ, seg_pos={0 0}, upload=56042]: Done uploading piece. err: <nil> Jul 26 21:59:33.542587183 (58h23m43.265150253s) [noise=true , address=45.91.116.8:4048 , piece_id=OAJ4H7OOUQU6L5T2PBY7B375ZOPRHA425C2YI5SK7HEE4VXLVR4Q, seg_pos={0 0}, upload=56042]: Done uploading piece. err: <nil> Jul 26 21:59:33.595897191 (58h23m43.318460231s) [noise=true , address=45.91.116.13:4154 , piece_id=2RJKH27HHTUL32USKWGHY5CRXIMYJTEIERZ3DL2TBSHS4I2COWNQ, seg_pos={0 0}, upload=56042]: Done uploading piece. err: <nil> Jul 26 21:59:33.620804146 (58h23m43.343367186s) [noise=true , address=45.91.116.5:4297 , piece_id=542GSYWXEKL57Z35LWY2G3FKNBHSIAOX773CSR5EBRZJGBQEVYQQ, seg_pos={0 0}, upload=56042]: Done uploading piece. err: <nil> Jul 26 21:59:33.622997563 (58h23m43.345560593s) [noise=true , address=45.91.116.10:4081 , piece_id=PCIFS5JSU5UJGUBAAEUU5OWTTQK4D4OPWSUEYARXIRNVANONIR2Q, seg_pos={0 0}, upload=56042]: Done uploading piece. err: <nil> Jul 26 21:59:33.651619846 (58h23m43.374182886s) [noise=true , address=45.91.116.6:4973 , piece_id=KBXWDUZO3GRJFI5O6K4SSQPOQBTFV4CXQJSDYC6QEM5AMY5H5SMA, seg_pos={0 0}, upload=56042]: Done uploading piece. err: <nil> Jul 26 21:59:33.672224078 (58h23m43.394787118s) [noise=true , address=45.91.116.12:4226 , piece_id=MGDXGUZ3YMWANOW2O77VWXBXENUPJTJZMCDBQ4D3GPQNO7STAEJQ, seg_pos={0 0}, upload=56042]: Done uploading piece. err: <nil> Jul 26 21:59:33.685439704 (58h23m43.408002734s) [noise=true , address=45.91.116.13:4153 , piece_id=2X4PGTRSD3OSOKORNDCRL4C7FP5IEBX5LWLYMHLZSFCYPR2L5PAA, seg_pos={0 0}, upload=56042]: Done uploading piece. err: <nil> Jul 26 22:00:00.619035194 (58h24m10.341598234s) [noise=true , address=104.36.82.2:4730 , piece_id=ETEMO7KOSUDPD37XNKYFOXOKV7PHZIFAVU4X7KLF7WNZOU56PFAQ, seg_pos={0 0}, upload=56042]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: rpc: dial tcp 104.36.82.2:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:00:00.619085841 (58h24m10.341648871s) [noise=true , address=104.36.82.16:4295 , piece_id=3QGMQQ4QPG6S5DB6I26BCVCX3NL4Q2W6VRCPH34SRC7XO6NZ6EHA, seg_pos={0 0}, upload=56042]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:43378->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:00:00.619092453 (58h24m10.341655493s) [noise=true , address=77.90.31.19:4842 , piece_id=CVXEA2ET3W62HQS6652SZZNPBXWD3E5HJH6VV7HQXL5O2VUCC4FA, seg_pos={0 0}, upload=56042]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:00:00.619122010 (58h24m10.34168505s) [noise=true , address=104.36.82.9:4007 , piece_id=TN35FQD6U5HWE6DEIPW2UGTCY4EVPA2F7GHLY5KQ5NTKO7Y3F5XA, seg_pos={0 0}, upload=56042]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 188.166.225.170:34290->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:00:00.620158793 (58h24m10.342721834s) [noise=true , address=104.36.82.4:4802 , piece_id=DISWA4JHXGUS7TUPFYUJUQ2KK4W4CCUB4K62UNHJRNQCPFJZSGRQ, seg_pos={0 0}, upload=56042]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:00:00.620197447 (58h24m10.342760487s) [seg_pos={0 0}, upload=56042]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 22:00:00.852187346 (58h24m10.574750386s) [upload=56042]: Done uploading segments... Jul 26 22:00:00.852213657 (58h24m10.574776697s) [upload=56042]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.234652284s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [....X.................X.....................................] duration: 1.858132263s FAIL (2025-07-26 21:59:56 UTC) orbiter warning: digest mismatch: after 0783c6a051cf3b535b7c61f8e6f0cfe751970c7121d9bce2182fd51e29ca8fc6 != before ee59801cb45aa958945ed73fcee20d9c3604e7f558c75f49df119aca36a29d88 Duration: 1.858164114s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X..X....] duration: 16.957462503s FAIL (2025-07-26 21:59:44 UTC) orbiter warning: context deadline exceeded Duration: 16.957487772s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 18 out of 60 is failed (threshold=3) [............................XX.........XX.XX.XXXX.XX.XXXXX.X] duration: 30.00238353s FAIL (2025-07-26 21:59:29 UTC) orbiter warning: uplink: context deadline exceeded Duration: 30.002408608s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....X.................X....................................] duration: 1.835356485s FAIL (2025-07-26 21:59:27 UTC) orbiter warning: digest mismatch: after 0783c6a051cf3b535b7c61f8e6f0cfe751970c7121d9bce2182fd51e29ca8fc6 != before ee59801cb45aa958945ed73fcee20d9c3604e7f558c75f49df119aca36a29d88 Duration: 1.835384489s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 30 out of 60 is failed (threshold=3) [..............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.235193026s FAIL (2025-07-26 21:59:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== JJ2IDSQ, seg_pos={0 0}, upload=56024]: Uploading piece... Jul 26 21:58:25.141866069 (58h22m34.864429109s) [noise=true , address=104.36.82.46:4488 , piece_id=QPGI4OV3AHJCTRSDSJETLFAWW5VJJKNIT72QJDLJONNJY5HD7Q4Q, seg_pos={0 0}, upload=56024]: Uploading piece... Jul 26 21:58:25.142084297 (58h22m34.864647337s) [noise=true , address=79.127.205.234:4605 , piece_id=UDIJPL6RQHZJGEMIZ4XTRIGMP7Q3MCJH7SHX6ZKZTDLOHPSRQMQQ, seg_pos={0 0}, upload=56024]: Uploading piece... Jul 26 21:58:25.142283848 (58h22m34.864846888s) [noise=true , address=104.36.82.47:4530 , piece_id=YRT5B3GTUDNTZK42VVZ3DPVAYR7QYERH5AXHVTT6PCAEUK5HGF2A, seg_pos={0 0}, upload=56024]: Uploading piece... Jul 26 21:58:25.142487829 (58h22m34.865050859s) [noise=true , address=64.49.13.170:4718 , piece_id=VUSPGCLW2BPDK6QUVFQ6DLUCLPPQW4ZHXKYZV4ZDTYKSLNG6EUFA, seg_pos={0 0}, upload=56024]: Uploading piece... Jul 26 21:58:25.142496005 (58h22m34.865059045s) [noise=true , address=104.36.82.41:4793 , piece_id=3SWU7FE34A3BU3KRJ3AUN7F3AJMHIUFN7SJX2WZFLE6OMFWXN44Q, seg_pos={0 0}, upload=56024]: Uploading piece... Jul 26 21:58:25.142672843 (58h22m34.865235873s) [noise=true , address=104.36.82.56:4264 , piece_id=J6U6RV6X6WTETJEGCJMC6GS4LVNTIS7CVKYJIJVBFYAOUAEIJE3A, seg_pos={0 0}, upload=56024]: Uploading piece... Jul 26 21:58:25.143060986 (58h22m34.865624026s) [noise=true , address=104.36.82.49:4000 , piece_id=LKWT6QKX5IHF62XRJKYH4LQHE6THVU2IDVUPQKUNJBTBGJ2EZLUA, seg_pos={0 0}, upload=56024]: Uploading piece... Jul 26 21:58:25.144168515 (58h22m34.866731555s) [noise=true , address=104.36.82.54:4165 , piece_id=4R4NSITL3XCUMTZQ6LMWK7OZCMEKAPVBTMW5WCQ7AX62PR4YRALQ, seg_pos={0 0}, upload=56024]: Uploading piece... Jul 26 21:58:25.145155303 (58h22m34.867718343s) [noise=true , address=104.36.82.6:4887 , piece_id=YERRO4DOH6K2FJXL4GENAGAJZEMOVUDBECOBUURHZJTHRYTASXIA, seg_pos={0 0}, upload=56024]: Uploading piece... Jul 26 21:58:25.146827923 (58h22m34.869390963s) [noise=true , address=104.36.82.52:4090 , piece_id=2CFUO64JSWDQAXAS3BASZTVLB37QRQCDJQWK2GBAFNR5DDXLGRFQ, seg_pos={0 0}, upload=56024]: Uploading piece... Jul 26 21:58:25.148414067 (58h22m34.870977107s) [noise=true , address=104.36.82.6:4899 , piece_id=HRJOAFW5GW44BYEJ2PFDFM3VMCQGA6DD5RJKRCASOK7FARLSF3PQ, seg_pos={0 0}, upload=56024]: Uploading piece... Jul 26 21:58:25.150012737 (58h22m34.872575787s) [noise=true , address=104.36.82.56:4233 , piece_id=SIIMGBUJSYYMVT6ZZAOVQHCTCLY6PV4IZ6EBIHZJRR2ZS6TF22LQ, seg_pos={0 0}, upload=56024]: Uploading piece... Jul 26 21:58:25.151578343 (58h22m34.874141383s) [noise=true , address=104.36.82.46:4505 , piece_id=G7HQVDERN4XROV66NBNOJ47NGFGZ5T7TOBIDPTFZBOK754IUWBPQ, seg_pos={0 0}, upload=56024]: Uploading piece... Jul 26 21:58:26.287264350 (58h22m36.00982738s) [upload=56024]: Next returned nil segment Jul 26 21:58:26.287280361 (58h22m36.009843391s) [upload=56024]: Waiting for error group managing segments... Jul 26 21:58:26.590502047 (58h22m36.313065088s) [noise=true , address=79.127.205.235:4685 , piece_id=HOVP2XMS7ZPVYOBS4YYVKWSNDSE7O36XZZP5QJXCLMI2IVAJ5PYA, seg_pos={0 0}, upload=56024]: Done uploading piece. err: <nil> Jul 26 21:58:26.613127437 (58h22m36.335690477s) [noise=true , address=79.127.205.234:4605 , piece_id=UDIJPL6RQHZJGEMIZ4XTRIGMP7Q3MCJH7SHX6ZKZTDLOHPSRQMQQ, seg_pos={0 0}, upload=56024]: Done uploading piece. err: <nil> Jul 26 21:58:27.339009644 (58h22m37.061572684s) [noise=true , address=79.127.219.38:4010 , piece_id=B5FSXUKTGUML764N4NY46KUIP67V3KHLML5WO55MS4JIEJJ2IDSQ, seg_pos={0 0}, upload=56024]: Done uploading piece. err: <nil> Jul 26 21:58:30.965331789 (58h22m40.687894859s) [noise=true , address=64.49.13.170:4718 , piece_id=VUSPGCLW2BPDK6QUVFQ6DLUCLPPQW4ZHXKYZV4ZDTYKSLNG6EUFA, seg_pos={0 0}, upload=56024]: Done uploading piece. err: <nil> Jul 26 21:58:32.344699146 (58h22m42.067262186s) [noise=true , address=79.127.201.215:4359 , piece_id=SRV3C66LMFMZWK6HPW74GFIAUAWLCDNTNIZUMNQJF5D7QHFN6DXQ, seg_pos={0 0}, upload=56024]: Done uploading piece. err: <nil> Jul 26 21:58:49.280564649 (58h22m59.003127689s) [noise=true , address=77.90.31.12:4608 , piece_id=TKO7UGK4VC6ILFB3EVZ33FSCGFBBDOD7MBNOQZJENKDEKWIBC7WA, seg_pos={0 0}, upload=56024]: Done uploading piece. err: <nil> Jul 26 21:59:24.038419522 (58h23m33.760982562s) [noise=true , address=104.36.82.46:4488 , piece_id=QPGI4OV3AHJCTRSDSJETLFAWW5VJJKNIT72QJDLJONNJY5HD7Q4Q, seg_pos={0 0}, upload=56024]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:104.36.82.46:4488): protocol: write tcp 188.166.225.170:59944->104.36.82.46:4488: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:59:24.038485248 (58h23m33.761048288s) [noise=true , address=104.36.82.41:4793 , piece_id=3SWU7FE34A3BU3KRJ3AUN7F3AJMHIUFN7SJX2WZFLE6OMFWXN44Q, seg_pos={0 0}, upload=56024]: Done uploading piece. err: ecclient: upload failed (node:1asDxxBjiU6yNbZ5c6uBmSWQ4QbVs5J3ErLbpKBSwqaptpNhpn, address:104.36.82.41:4793): protocol: rpc: dial tcp 104.36.82.41:4793: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:59:24.038567015 (58h23m33.761130055s) [noise=true , address=104.36.82.54:4165 , piece_id=4R4NSITL3XCUMTZQ6LMWK7OZCMEKAPVBTMW5WCQ7AX62PR4YRALQ, seg_pos={0 0}, upload=56024]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: rpc: dial tcp 104.36.82.54:4165: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:59:24.038623152 (58h23m33.761186192s) [noise=true , address=104.36.82.49:4000 , piece_id=LKWT6QKX5IHF62XRJKYH4LQHE6THVU2IDVUPQKUNJBTBGJ2EZLUA, seg_pos={0 0}, upload=56024]: Done uploading piece. err: ecclient: upload failed (node:12chhfe1weiPC1jjv8vbScP7uFVNXFr6s3UcNRkH8gsSAtGD16D, address:104.36.82.49:4000): protocol: rpc: dial tcp 104.36.82.49:4000: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:59:24.038652989 (58h23m33.761216029s) [noise=true , address=104.36.82.6:4899 , piece_id=HRJOAFW5GW44BYEJ2PFDFM3VMCQGA6DD5RJKRCASOK7FARLSF3PQ, seg_pos={0 0}, upload=56024]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: rpc: dial tcp 104.36.82.6:4899: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:59:24.038685391 (58h23m33.761248431s) [noise=true , address=104.36.82.56:4233 , piece_id=SIIMGBUJSYYMVT6ZZAOVQHCTCLY6PV4IZ6EBIHZJRR2ZS6TF22LQ, seg_pos={0 0}, upload=56024]: Done uploading piece. err: ecclient: upload failed (node:1DmQeoQWTnFhQeJFseJoNYzFyo7vZjYKwDtQVJ6pomksWExsTb, address:104.36.82.56:4233): protocol: rpc: dial tcp 104.36.82.56:4233: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:59:24.038740266 (58h23m33.761303296s) [noise=true , address=104.36.82.47:4530 , piece_id=YRT5B3GTUDNTZK42VVZ3DPVAYR7QYERH5AXHVTT6PCAEUK5HGF2A, seg_pos={0 0}, upload=56024]: Done uploading piece. err: ecclient: upload failed (node:1WVxMFx9jm6CT7GcXg2qgq3Be9UPFf1KZKssqufSk2ebdBBmxR, address:104.36.82.47:4530): protocol: write tcp 188.166.225.170:35380->104.36.82.47:4530: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:59:24.038775072 (58h23m33.761338102s) [noise=true , address=104.36.82.56:4264 , piece_id=J6U6RV6X6WTETJEGCJMC6GS4LVNTIS7CVKYJIJVBFYAOUAEIJE3A, seg_pos={0 0}, upload=56024]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:59:24.038872128 (58h23m33.761435169s) [noise=true , address=104.36.82.6:4887 , piece_id=YERRO4DOH6K2FJXL4GENAGAJZEMOVUDBECOBUURHZJTHRYTASXIA, seg_pos={0 0}, upload=56024]: Done uploading piece. err: ecclient: upload failed (node:19QrU8gBXbSzy9LzhkHTYgdtTbZG51wV9CSWRcvvQuM6Jex9d7, address:104.36.82.6:4887): protocol: rpc: dial tcp 104.36.82.6:4887: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:59:24.038905151 (58h23m33.761468192s) [noise=true , address=104.36.82.52:4090 , piece_id=2CFUO64JSWDQAXAS3BASZTVLB37QRQCDJQWK2GBAFNR5DDXLGRFQ, seg_pos={0 0}, upload=56024]: Done uploading piece. err: ecclient: upload failed (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F, address:104.36.82.52:4090): protocol: rpc: dial tcp 104.36.82.52:4090: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:59:24.038966048 (58h23m33.761529088s) [noise=true , address=104.36.82.46:4505 , piece_id=G7HQVDERN4XROV66NBNOJ47NGFGZ5T7TOBIDPTFZBOK754IUWBPQ, seg_pos={0 0}, upload=56024]: Done uploading piece. err: ecclient: upload failed (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf, address:104.36.82.46:4505): protocol: rpc: dial tcp 104.36.82.46:4505: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:59:24.039003490 (58h23m33.76156653s) [noise=true , address=104.36.82.55:4218 , piece_id=JIANCANNSUSO3ERCTBOPAJQHW3SD2OL7RQADIRTLN4UEEETYOV6Q, seg_pos={0 0}, upload=56024]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: rpc: dial tcp 104.36.82.55:4218: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:59:24.039031614 (58h23m33.761594644s) [seg_pos={0 0}, upload=56024]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 26 21:59:24.271511780 (58h23m33.99407482s) [upload=56024]: Done uploading segments... Jul 26 21:59:24.271540826 (58h23m33.994103866s) [upload=56024]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1m0.235238882s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X..X...] duration: 5.908770655s FAIL (2025-07-26 21:59:17 UTC) orbiter warning: context deadline exceeded Duration: 5.908792976s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [......................X....X......XX....XX.XX.XXX.X.XX.XXX.X] duration: 30.229092728s FAIL (2025-07-26 21:58:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 56951488s) [upload=56008]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 21:57:41.717271330 (58h21m51.43983437s) [upload=56008]: Done uploading segments... Jul 26 21:57:41.717284696 (58h21m51.439847736s) [upload=56008]: Upload finished. err: <nil> Jul 26 21:58:17.517633033 (58h22m27.240196073s) [upload=56022]: Starting upload Jul 26 21:58:17.517724487 (58h22m27.240287527s) [upload=56022]: Done starting upload Jul 26 21:58:17.517966711 (58h22m27.240529751s) [upload=56022]: Uploading segments... Jul 26 21:58:17.518002088 (58h22m27.240565128s) [upload=56022]: Got next segment. Inline: false Jul 26 21:58:17.518028409 (58h22m27.240591459s) [upload=56022]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 21:58:17.518033739 (58h22m27.240596779s) [upload=56022]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 21:58:17.748435269 (58h22m27.470998309s) [seg_pos={0 0}, upload=56022]: Begin upload segment... Jul 26 21:58:17.748475305 (58h22m27.471038345s) [seg_pos={0 0}, upload=56022]: Done begin upload segment. Jul 26 21:58:17.748494231 (58h22m27.471057271s) [noise=true , address=45.91.116.4:4938 , piece_id=BB27OAWNJGUBBXHUDK3NZG2WE4UGJKPMM7A4H5AEY3XFFWROUDOA, seg_pos={0 0}, upload=56022]: Uploading piece... Jul 26 21:58:17.748520632 (58h22m27.471083672s) [noise=true , address=45.91.116.14:4189 , piece_id=A5KC7SV5C26YMMMPLK6DQFQQ4QVXOW4NNPZOTCYZXB4ESLJPJSWA, seg_pos={0 0}, upload=56022]: Uploading piece... Jul 26 21:58:17.748837739 (58h22m27.471400769s) [noise=true , address=45.91.116.12:4226 , piece_id=EF7VG5DG5IM6MIU2R6I32PWD3G5QURHOHZYSM6ELGLX42XA7AEDQ, seg_pos={0 0}, upload=56022]: Uploading piece... Jul 26 21:58:17.748877735 (58h22m27.471440775s) [noise=true , address=77.90.31.16:4734 , piece_id=DTFODWBGBB6DLQG4HQWOIHTRHYXW74JC4A2HDD5T57XZICVQ7KGA, seg_pos={0 0}, upload=56022]: Uploading piece... Jul 26 21:58:17.749111924 (58h22m27.471674964s) [noise=true , address=45.91.116.7:4010 , piece_id=UUSR35WPPKZ72T4SVN65O67DH6MKDMBMTJTO6ZHPKXXTOELPKPOA, seg_pos={0 0}, upload=56022]: Uploading piece... Jul 26 21:58:17.749118917 (58h22m27.471681957s) [noise=true , address=104.36.82.8:4970 , piece_id=HTN6FFOYG5KVMLGN5SKOCQGSOZAMGBKFPRVYB6XCSJIKHSCCBTMA, seg_pos={0 0}, upload=56022]: Uploading piece... Jul 26 21:58:17.749348646 (58h22m27.471911686s) [noise=true , address=45.91.116.6:4975 , piece_id=TGVYOWJK7HHGPMOJMCLQE62GSNLORKCAPLN2PO6732ZJCVI3ERNA, seg_pos={0 0}, upload=56022]: Uploading piece... Jul 26 21:58:17.749378884 (58h22m27.471941924s) [noise=true , address=45.91.116.9:4262 , piece_id=5BL5HZKFIPNDTWZQIUYMJQAMXPGM6CQTHDS7BZM4IKMLE3ESENJA, seg_pos={0 0}, upload=56022]: Uploading piece... Jul 26 21:58:17.749547927 (58h22m27.472110967s) [noise=true , address=45.91.116.11:4119 , piece_id=F5IHGE25DZEYLW5ROYUGDGQ74UZHKFF4BYE2ERYAAUX37E5ZHATA, seg_pos={0 0}, upload=56022]: Uploading piece... Jul 26 21:58:17.749648931 (58h22m27.472211961s) [noise=true , address=104.36.82.12:4150 , piece_id=6F5ZER2Q2E27ODKIVN4MQVSYB5XSPYCJCI6UMZSQRDTNLNLO5IAA, seg_pos={0 0}, upload=56022]: Uploading piece... Jul 26 21:58:17.749748652 (58h22m27.472311692s) [noise=true , address=77.90.31.10:4517 , piece_id=6NOCXY3U2IQUSOU2Y6BF3UAQ6FU6VS55Y5PYYPZ2KUZ5WLIWZATQ, seg_pos={0 0}, upload=56022]: Uploading piece... Jul 26 21:58:17.749924798 (58h22m27.472487838s) [noise=true , address=104.36.82.3:4765 , piece_id=TL2TEJHCC2MWBNYDY3HQXXK4DQ5TVEHIWK7LDWOQZPW3LXILTC7A, seg_pos={0 0}, upload=56022]: Uploading piece... Jul 26 21:58:17.750136674 (58h22m27.472699714s) [noise=true , address=104.36.82.2:4732 , piece_id=YQW3VMB4UYWRSKH4WNQY7WMBCFYRWKTDR62FRKGG7FTGJ3IYCISA, seg_pos={0 0}, upload=56022]: Uploading piece... Jul 26 21:58:17.750336035 (58h22m27.472899075s) [noise=true , address=104.36.82.9:4006 , piece_id=QB3USNHTYXLRY6BR2XFUQVJ725JMOF4LC4N4624T2XZDHK4DKXIQ, seg_pos={0 0}, upload=56022]: Uploading piece... Jul 26 21:58:17.750563480 (58h22m27.47312651s) [noise=true , address=77.90.31.10:4520 , piece_id=2GCWNIBY2GFLC7ZSDIDSJ277VMGGEJIX4K6JP4VURAK57CTZ6SEQ, seg_pos={0 0}, upload=56022]: Uploading piece... Jul 26 21:58:17.751115476 (58h22m27.473678516s) [noise=true , address=45.91.116.13:4154 , piece_id=UR4I4ORFBOHWVQIQNZTQ56J6D65YI53O7A6UEG2UPUSR6LGEEUQQ, seg_pos={0 0}, upload=56022]: Uploading piece... Jul 26 21:58:19.520527338 (58h22m29.243090378s) [upload=56022]: Next returned nil segment Jul 26 21:58:19.520544210 (58h22m29.24310725s) [upload=56022]: Waiting for error group managing segments... Jul 26 21:58:19.857939662 (58h22m29.580502712s) [noise=true , address=45.91.116.11:4119 , piece_id=F5IHGE25DZEYLW5ROYUGDGQ74UZHKFF4BYE2ERYAAUX37E5ZHATA, seg_pos={0 0}, upload=56022]: Done uploading piece. err: <nil> Jul 26 21:58:19.871741239 (58h22m29.594304289s) [noise=true , address=45.91.116.6:4975 , piece_id=TGVYOWJK7HHGPMOJMCLQE62GSNLORKCAPLN2PO6732ZJCVI3ERNA, seg_pos={0 0}, upload=56022]: Done uploading piece. err: <nil> Jul 26 21:58:19.872002839 (58h22m29.594565879s) [noise=true , address=45.91.116.7:4010 , piece_id=UUSR35WPPKZ72T4SVN65O67DH6MKDMBMTJTO6ZHPKXXTOELPKPOA, seg_pos={0 0}, upload=56022]: Done uploading piece. err: <nil> Jul 26 21:58:19.906813954 (58h22m29.629377004s) [noise=true , address=45.91.116.12:4226 , piece_id=EF7VG5DG5IM6MIU2R6I32PWD3G5QURHOHZYSM6ELGLX42XA7AEDQ, seg_pos={0 0}, upload=56022]: Done uploading piece. err: <nil> Jul 26 21:58:19.910889391 (58h22m29.633452431s) [noise=true , address=45.91.116.4:4938 , piece_id=BB27OAWNJGUBBXHUDK3NZG2WE4UGJKPMM7A4H5AEY3XFFWROUDOA, seg_pos={0 0}, upload=56022]: Done uploading piece. err: <nil> Jul 26 21:58:19.917549604 (58h22m29.640112644s) [noise=true , address=45.91.116.14:4189 , piece_id=A5KC7SV5C26YMMMPLK6DQFQQ4QVXOW4NNPZOTCYZXB4ESLJPJSWA, seg_pos={0 0}, upload=56022]: Done uploading piece. err: <nil> Jul 26 21:58:19.919745786 (58h22m29.642308826s) [noise=true , address=45.91.116.9:4262 , piece_id=5BL5HZKFIPNDTWZQIUYMJQAMXPGM6CQTHDS7BZM4IKMLE3ESENJA, seg_pos={0 0}, upload=56022]: Done uploading piece. err: <nil> Jul 26 21:58:19.928585079 (58h22m29.651148119s) [noise=true , address=45.91.116.13:4154 , piece_id=UR4I4ORFBOHWVQIQNZTQ56J6D65YI53O7A6UEG2UPUSR6LGEEUQQ, seg_pos={0 0}, upload=56022]: Done uploading piece. err: <nil> Jul 26 21:58:24.122356453 (58h22m33.844919493s) [noise=true , address=77.90.31.16:4734 , piece_id=DTFODWBGBB6DLQG4HQWOIHTRHYXW74JC4A2HDD5T57XZICVQ7KGA, seg_pos={0 0}, upload=56022]: Done uploading piece. err: <nil> Jul 26 21:58:28.538402874 (58h22m38.260965904s) [noise=true , address=77.90.31.10:4520 , piece_id=2GCWNIBY2GFLC7ZSDIDSJ277VMGGEJIX4K6JP4VURAK57CTZ6SEQ, seg_pos={0 0}, upload=56022]: Done uploading piece. err: <nil> Jul 26 21:58:46.879326578 (58h22m56.601889618s) [noise=true , address=104.36.82.3:4765 , piece_id=TL2TEJHCC2MWBNYDY3HQXXK4DQ5TVEHIWK7LDWOQZPW3LXILTC7A, seg_pos={0 0}, upload=56022]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:58:46.879385621 (58h22m56.601948661s) [noise=true , address=77.90.31.10:4517 , piece_id=6NOCXY3U2IQUSOU2Y6BF3UAQ6FU6VS55Y5PYYPZ2KUZ5WLIWZATQ, seg_pos={0 0}, upload=56022]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:53314->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:53314->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53314->77.90.31.10:4517: use of closed network connection Jul 26 21:58:46.879421249 (58h22m56.601984289s) [noise=true , address=104.36.82.9:4006 , piece_id=QB3USNHTYXLRY6BR2XFUQVJ725JMOF4LC4N4624T2XZDHK4DKXIQ, seg_pos={0 0}, upload=56022]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 188.166.225.170:50606->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:58:46.879444864 (58h22m56.602007904s) [noise=true , address=104.36.82.2:4732 , piece_id=YQW3VMB4UYWRSKH4WNQY7WMBCFYRWKTDR62FRKGG7FTGJ3IYCISA, seg_pos={0 0}, upload=56022]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:58:46.879448932 (58h22m56.602011972s) [noise=true , address=104.36.82.12:4150 , piece_id=6F5ZER2Q2E27ODKIVN4MQVSYB5XSPYCJCI6UMZSQRDTNLNLO5IAA, seg_pos={0 0}, upload=56022]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 188.166.225.170:49390->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:58:46.879477627 (58h22m56.602040667s) [noise=true , address=104.36.82.8:4970 , piece_id=HTN6FFOYG5KVMLGN5SKOCQGSOZAMGBKFPRVYB6XCSJIKHSCCBTMA, seg_pos={0 0}, upload=56022]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:58:46.879499539 (58h22m56.602062579s) [seg_pos={0 0}, upload=56022]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 26 21:58:47.106507134 (58h22m56.829070174s) [upload=56022]: Done uploading segments... Jul 26 21:58:47.106530378 (58h22m56.829093418s) [upload=56022]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 30.229116554s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 29 out of 60 is failed (threshold=3) [...............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.228856643s FAIL (2025-07-26 21:58:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 4) ======LOGS====== FAJDIAN3DQQOWTHLB6CQ, seg_pos={0 0}, upload=56005]: Uploading piece... Jul 26 21:57:21.101277956 (58h21m30.823840986s) [noise=true , address=104.36.82.50:4015 , piece_id=IJCHPEEZNEJYQEJNZLRZTDYXHQQBYL7UY7OGW2GYPRQHWZZYG7WQ, seg_pos={0 0}, upload=56005]: Uploading piece... Jul 26 21:57:21.101532122 (58h21m30.824095162s) [noise=true , address=104.36.82.54:4192 , piece_id=PTY2P7F6JEVNL4RBCN47WQ7G2LQKUD774Y2AXHZIU2S2MVAUMKCA, seg_pos={0 0}, upload=56005]: Uploading piece... Jul 26 21:57:21.101573761 (58h21m30.824136801s) [noise=true , address=104.36.82.56:4260 , piece_id=U3MKTYNRCBNQQWEEDNKYVDWLYAYOWXO26KPI3OGBXIMWTWSYC47A, seg_pos={0 0}, upload=56005]: Uploading piece... Jul 26 21:57:21.101913963 (58h21m30.824477003s) [noise=true , address=104.36.82.19:4414 , piece_id=IFHPUUJUZ2VRAKD7L3PTRPYJHK6PPGT432O3VMLFFGDNSHVFWXRA, seg_pos={0 0}, upload=56005]: Uploading piece... Jul 26 21:57:21.101922479 (58h21m30.824485509s) [noise=true , address=104.36.82.53:4152 , piece_id=WERKLUTZVG7ULMGO3XG267C7YAPCVMKKIMZZ74X4GE6LVUEZMS4A, seg_pos={0 0}, upload=56005]: Uploading piece... Jul 26 21:57:21.102172357 (58h21m30.824735397s) [noise=true , address=79.127.201.215:4354 , piece_id=6KHHQBFCWMY7DUY7B574IYKD37O2PQWG3WYLQAQICGVR2UIV7ZQQ, seg_pos={0 0}, upload=56005]: Uploading piece... Jul 26 21:57:21.102467351 (58h21m30.825030391s) [noise=true , address=104.36.82.47:4534 , piece_id=N54XMDZTOZBDJTLRDLX7BKYW46YUEOZ67ZSBTO5EI3ZTEDHF2XQQ, seg_pos={0 0}, upload=56005]: Uploading piece... Jul 26 21:57:21.102706309 (58h21m30.825269349s) [noise=true , address=104.36.82.20:4454 , piece_id=5LC7FMGTT3YDX6FO5SXXCXPX46KWDO3N6R4XHFKW45KG4UR3GK7A, seg_pos={0 0}, upload=56005]: Uploading piece... Jul 26 21:57:21.103636578 (58h21m30.826199618s) [noise=true , address=104.36.82.12:4173 , piece_id=OW37MCQVLTOPQHZSVLCV6PQVB5ALRWBMPOKTCRKFL4NMUF4NZQ3Q, seg_pos={0 0}, upload=56005]: Uploading piece... Jul 26 21:57:22.207329885 (58h21m31.929892925s) [upload=56005]: Next returned nil segment Jul 26 21:57:22.207350144 (58h21m31.929913184s) [upload=56005]: Waiting for error group managing segments... Jul 26 21:57:22.505917737 (58h21m32.228480767s) [noise=true , address=79.127.205.235:4668 , piece_id=TAWLJZ2ZJS5T47G3YJBRD4GPDL67TH4ECMEJCW6NZ5HKUUOYXL6Q, seg_pos={0 0}, upload=56005]: Done uploading piece. err: <nil> Jul 26 21:57:22.556364187 (58h21m32.278927227s) [noise=true , address=79.127.205.246:4279 , piece_id=H4MSLNWJYMUQMLFVQE7FMDYMBEZMWJET7MOQWUXAEC5NUC4JGMMA, seg_pos={0 0}, upload=56005]: Done uploading piece. err: <nil> Jul 26 21:57:23.057357212 (58h21m32.779920262s) [noise=true , address=79.127.219.38:4018 , piece_id=PL3GFIWVACH75YRM6O52XL54T2DSCDF5DG6MZEKGPICCRYK2LVWQ, seg_pos={0 0}, upload=56005]: Done uploading piece. err: <nil> Jul 26 21:57:26.777259521 (58h21m36.499822551s) [noise=true , address=79.127.201.215:4354 , piece_id=6KHHQBFCWMY7DUY7B574IYKD37O2PQWG3WYLQAQICGVR2UIV7ZQQ, seg_pos={0 0}, upload=56005]: Done uploading piece. err: <nil> Jul 26 21:58:20.012763438 (58h22m29.735326478s) [noise=true , address=104.36.82.6:4902 , piece_id=AFZYFZN4KAFIB63PGJ7PEOWSZ23KDHI6LR2GL3IPOIFYT56XJZVQ, seg_pos={0 0}, upload=56005]: Done uploading piece. err: ecclient: upload failed (node:12cXPznDWzr4KDxdxZDEoCpubdfEeMSdyyemGMhXcFTycSwatyv, address:104.36.82.6:4902): protocol: rpc: dial tcp 104.36.82.6:4902: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:58:20.012867066 (58h22m29.735430107s) [noise=true , address=104.36.82.16:4306 , piece_id=HWYK27BTVPZMDQR44CJEFQZ3GRYT2S34FAJDIAN3DQQOWTHLB6CQ, seg_pos={0 0}, upload=56005]: Done uploading piece. err: ecclient: upload failed (node:1mcaP7H2gd9gWpbcPK79gvemz5L6k5UgbwQ5t6aaTU7nXut1p7, address:104.36.82.16:4306): protocol: rpc: dial tcp 104.36.82.16:4306: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:58:20.012918394 (58h22m29.735481434s) [noise=true , address=104.36.82.19:4414 , piece_id=IFHPUUJUZ2VRAKD7L3PTRPYJHK6PPGT432O3VMLFFGDNSHVFWXRA, seg_pos={0 0}, upload=56005]: Done uploading piece. err: ecclient: upload failed (node:1R6GJNS7bXfsDLsp3CE9qcvTbcT44QmgvSFzVbzpukyVYsezRQ, address:104.36.82.19:4414): protocol: write tcp 188.166.225.170:42552->104.36.82.19:4414: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:58:20.012952189 (58h22m29.735515229s) [noise=true , address=104.36.82.56:4260 , piece_id=U3MKTYNRCBNQQWEEDNKYVDWLYAYOWXO26KPI3OGBXIMWTWSYC47A, seg_pos={0 0}, upload=56005]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: write tcp 188.166.225.170:39938->104.36.82.56:4260: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:58:20.012980263 (58h22m29.735543293s) [noise=true , address=104.36.82.53:4152 , piece_id=WERKLUTZVG7ULMGO3XG267C7YAPCVMKKIMZZ74X4GE6LVUEZMS4A, seg_pos={0 0}, upload=56005]: Done uploading piece. err: ecclient: upload failed (node:127p3Fibbp6XQSewgrTiBAn45TSviFfjSGpv92tGLxsDSKWPuUe, address:104.36.82.53:4152): protocol: write tcp 188.166.225.170:33596->104.36.82.53:4152: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:58:20.013015220 (58h22m29.73557826s) [noise=true , address=104.36.82.50:4015 , piece_id=IJCHPEEZNEJYQEJNZLRZTDYXHQQBYL7UY7OGW2GYPRQHWZZYG7WQ, seg_pos={0 0}, upload=56005]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:104.36.82.50:4015): protocol: write tcp 188.166.225.170:44260->104.36.82.50:4015: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:58:20.013055737 (58h22m29.735618777s) [noise=true , address=104.36.82.20:4454 , piece_id=5LC7FMGTT3YDX6FO5SXXCXPX46KWDO3N6R4XHFKW45KG4UR3GK7A, seg_pos={0 0}, upload=56005]: Done uploading piece. err: ecclient: upload failed (node:1SafkJPqKHLwMaqyZdLq3yJ1xdjqDtWHFrUwynmegUVwEC7y5J, address:104.36.82.20:4454): protocol: write tcp 188.166.225.170:39644->104.36.82.20:4454: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:58:20.013066378 (58h22m29.735629418s) [noise=true , address=104.36.82.7:4922 , piece_id=B5BXEKNJXOM5J54PPJVXAROGK5QTFERTNCGFR2C7ZJC2Z7HOMOIA, seg_pos={0 0}, upload=56005]: Done uploading piece. err: ecclient: upload failed (node:12KrMgT4Pwj9UFE88q6tt65D7o6q68NhsmEXmifNmvzJ6eEkm8R, address:104.36.82.7:4922): protocol: write tcp 188.166.225.170:52592->104.36.82.7:4922: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:58:20.013092528 (58h22m29.735655568s) [noise=true , address=104.36.82.12:4173 , piece_id=OW37MCQVLTOPQHZSVLCV6PQVB5ALRWBMPOKTCRKFL4NMUF4NZQ3Q, seg_pos={0 0}, upload=56005]: Done uploading piece. err: ecclient: upload failed (node:12HbGUhwH2eNqY75sLQCDgkvJ1QSrJ2n4Qg8A5Z73dFXxRpVwSC, address:104.36.82.12:4173): protocol: write tcp 188.166.225.170:57692->104.36.82.12:4173: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:58:20.013127975 (58h22m29.735691005s) [noise=true , address=104.36.82.52:4103 , piece_id=YVMFAQXYGHFAXNU4OOCNOZ7NH3QEFG36F6XZ7K5RN6N6STP7UUOQ, seg_pos={0 0}, upload=56005]: Done uploading piece. err: ecclient: upload failed (node:1v6zXZuMmyL2MU8qb3q6gg1iwTYvndjzMjVpv8btEpeX9Lnqmu, address:104.36.82.52:4103): protocol: write tcp 188.166.225.170:52676->104.36.82.52:4103: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:58:20.013167020 (58h22m29.735730051s) [noise=true , address=104.36.82.47:4534 , piece_id=N54XMDZTOZBDJTLRDLX7BKYW46YUEOZ67ZSBTO5EI3ZTEDHF2XQQ, seg_pos={0 0}, upload=56005]: Done uploading piece. err: ecclient: upload failed (node:1VDrNGcCnaKEmG5NN4nex1JoZAgdCEfX5CdsKbxytfLfKzPbr, address:104.36.82.47:4534): protocol: write tcp 188.166.225.170:43270->104.36.82.47:4534: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:58:20.013175256 (58h22m29.735738286s) [noise=true , address=104.36.82.55:4220 , piece_id=LUJP4XHLUL6DVVIVHR2D6CVJXMMQTEJ6ZSX64WQ3K6QDA6WFENYA, seg_pos={0 0}, upload=56005]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:104.36.82.55:4220): protocol: write tcp 188.166.225.170:60402->104.36.82.55:4220: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:58:20.013223298 (58h22m29.735786338s) [noise=true , address=104.36.82.46:4513 , piece_id=N6BN7WYSAYECKKYQYZRM23IOLLG5MJ5GEWGJNOJ42CUKVXGWXAFA, seg_pos={0 0}, upload=56005]: Done uploading piece. err: ecclient: upload failed (node:12pyyW8Gdqg6j9YGMtCUBhFnoNXryvPMUy19x815dp2HV3Xqr8h, address:104.36.82.46:4513): protocol: write tcp 188.166.225.170:44934->104.36.82.46:4513: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:58:20.013287370 (58h22m29.73585041s) [noise=true , address=104.36.82.54:4192 , piece_id=PTY2P7F6JEVNL4RBCN47WQ7G2LQKUD774Y2AXHZIU2S2MVAUMKCA, seg_pos={0 0}, upload=56005]: Done uploading piece. err: ecclient: upload failed (node:12peSG85BzWEjq3FZ5xMqLE2gStDvy2dViRdYmDJz7Lq8yBfaKY, address:104.36.82.54:4192): protocol: rpc: dial tcp 104.36.82.54:4192: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:58:20.013322367 (58h22m29.735885417s) [seg_pos={0 0}, upload=56005]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil> Jul 26 21:58:20.239393562 (58h22m29.961956602s) [upload=56005]: Done uploading segments... Jul 26 21:58:20.239418009 (58h22m29.961981049s) [upload=56005]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4) Duration: 1m0.228869196s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X..X..] duration: 8.207491645s FAIL (2025-07-26 21:58:18 UTC) orbiter warning: context deadline exceeded Duration: 8.207513487s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 17 out of 60 is failed (threshold=3) [.............................XX.........XX.XX.XXXX.XX.XXXXX.] duration: 19.867981072s FAIL (2025-07-26 21:58:17 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 19.868011109s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [......X.................X...................................] duration: 1.854328055s FAIL (2025-07-26 21:58:16 UTC) orbiter warning: digest mismatch: after 0783c6a051cf3b535b7c61f8e6f0cfe751970c7121d9bce2182fd51e29ca8fc6 != before ee59801cb45aa958945ed73fcee20d9c3604e7f558c75f49df119aca36a29d88 Duration: 1.854361909s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 17 out of 60 is failed (threshold=3) [..............................XX.........XX.XX.XXXX.XX.XXXXX] duration: 30.239433135s FAIL (2025-07-26 21:57:55 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.23946154s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.......................X....X......XX....XX.XX.XXX.X.XX.XXX.] duration: 17.249838281s FAIL (2025-07-26 21:57:44 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== d Jul 26 21:56:21.880426452 (58h20m31.602989492s) [noise=true , address=104.36.82.8:4969 , piece_id=7JFRXSBU7C3FVWXI4A6XV7WCDZQAV7VJPC4UJ2O7NQFVT6D4IM6A, seg_pos={0 0}, upload=55985]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: context deadline exceeded; context deadline exceeded Jul 26 21:56:21.880444877 (58h20m31.603007917s) [seg_pos={0 0}, upload=55985]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 21:56:22.111166668 (58h20m31.833729708s) [upload=55985]: Done uploading segments... Jul 26 21:56:22.111209851 (58h20m31.833772891s) [upload=55985]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 26 21:56:23.019512381 (58h20m32.742075421s) [upload=55991]: Starting upload Jul 26 21:56:23.019569490 (58h20m32.74213253s) [upload=55991]: Done starting upload Jul 26 21:56:23.019678088 (58h20m32.742241128s) [upload=55991]: Uploading segments... Jul 26 21:56:23.019837513 (58h20m32.742400573s) [upload=55991]: Got next segment. Inline: false Jul 26 21:56:23.019881758 (58h20m32.742444838s) [upload=55991]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 21:56:23.019890524 (58h20m32.742453574s) [upload=55991]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 21:56:23.255552398 (58h20m32.978115438s) [seg_pos={0 0}, upload=55991]: Begin upload segment... Jul 26 21:56:23.255606421 (58h20m32.978169451s) [seg_pos={0 0}, upload=55991]: Done begin upload segment. Jul 26 21:56:23.255629285 (58h20m32.978192325s) [noise=true , address=45.91.116.6:4975 , piece_id=EDZ7SRCY7NNIWCHNMK5ENXKHJZTAVGLIU5UJX6EPNQBNBGZWDPOQ, seg_pos={0 0}, upload=55991]: Uploading piece... Jul 26 21:56:23.255656206 (58h20m32.978219236s) [noise=true , address=45.91.116.12:4227 , piece_id=FCQL5H52LUAZWOKIJWS6GUZANHD2E7YFJSPCABRORVQAVRXHX6ZA, seg_pos={0 0}, upload=55991]: Uploading piece... Jul 26 21:56:23.256097681 (58h20m32.978660721s) [noise=true , address=45.91.116.5:4298 , piece_id=ZTBV2MODSZJK4XJF5B4AYOFW6BINA26XVRQPYWWI24JVQCXH562A, seg_pos={0 0}, upload=55991]: Uploading piece... Jul 26 21:56:23.256426109 (58h20m32.978989149s) [noise=true , address=104.36.82.15:4258 , piece_id=YPGQT6YYV4E732W5DSB7NRCCZJCFEVC33NGO526N3Q5MYY3UWMMA, seg_pos={0 0}, upload=55991]: Uploading piece... Jul 26 21:56:23.256710143 (58h20m32.979273183s) [noise=true , address=45.91.116.6:4973 , piece_id=AVZ4VI6FJDY5443WB5GLQSVC4J74IKUKOLIEEZLVM4CC4X4P4B3Q, seg_pos={0 0}, upload=55991]: Uploading piece... Jul 26 21:56:23.256774927 (58h20m32.979337967s) [noise=true , address=45.91.116.11:4117 , piece_id=MDWEVNUPXCADXKDB24IWT4PDK4D62LOTZ6M4AFH76DYM4CWZRQZQ, seg_pos={0 0}, upload=55991]: Uploading piece... Jul 26 21:56:23.257064120 (58h20m32.97962716s) [noise=true , address=104.36.82.8:4969 , piece_id=VMB3FRYKZGXDAJFLR6GKAFUX7IM4V7BB55RHXMS7Q3ZG7MZVUUYQ, seg_pos={0 0}, upload=55991]: Uploading piece... Jul 26 21:56:23.257312285 (58h20m32.979875315s) [noise=true , address=77.90.31.15:4697 , piece_id=WJC4QDUAVNMJA7LKHVWRGDKXXYNYI53SJDD4WV4F4YZHKBCOCEYQ, seg_pos={0 0}, upload=55991]: Uploading piece... Jul 26 21:56:23.257611768 (58h20m32.980174808s) [noise=true , address=77.90.31.11:4555 , piece_id=M52GC3KWD7XO6FJC77ERY23TT3GYPPVSDAUTXICXWIDIXEFDADNQ, seg_pos={0 0}, upload=55991]: Uploading piece... Jul 26 21:56:23.257909758 (58h20m32.980472798s) [noise=true , address=77.90.31.10:4519 , piece_id=VBGTE6B73E7GIWTWRRBYDSK4FQCPQI2HQDJNXUA3LLNSNPADV6TQ, seg_pos={0 0}, upload=55991]: Uploading piece... Jul 26 21:56:23.257954583 (58h20m32.980517623s) [noise=true , address=45.91.116.4:4937 , piece_id=5Y5GFRONP23RJ6KUVUIVFUM37QMAUMKXIMMPW7YUHDCLHIERMASQ, seg_pos={0 0}, upload=55991]: Uploading piece... Jul 26 21:56:23.258203630 (58h20m32.98076667s) [noise=true , address=45.91.116.7:4009 , piece_id=332GMURJ4CKAAJ5RYOKM42NDVJ3IQH7ZYIQ2WK3SVCTQVL2RSIAQ, seg_pos={0 0}, upload=55991]: Uploading piece... Jul 26 21:56:23.258431867 (58h20m32.980994907s) [noise=true , address=77.90.31.14:4661 , piece_id=UERZICXUIB2BY3YE77K3N7QUJ6S3Y3MNFFH23CN5DVCYSPLX7F7Q, seg_pos={0 0}, upload=55991]: Uploading piece... Jul 26 21:56:23.258523622 (58h20m32.981086662s) [noise=true , address=45.91.116.10:4083 , piece_id=LKTYNKD7SPUQPIC5CR4RWDRPVFS3TTIBBFARLAYYLGKFJHDZA24Q, seg_pos={0 0}, upload=55991]: Uploading piece... Jul 26 21:56:23.258820911 (58h20m32.981383951s) [noise=true , address=77.90.31.19:4843 , piece_id=56B4KRMMKSR7ZWS3RGT57PPOFRHVJBWXBC36HMQMOYKQWBMKIT2Q, seg_pos={0 0}, upload=55991]: Uploading piece... Jul 26 21:56:23.258976820 (58h20m32.98153985s) [noise=true , address=45.91.116.11:4118 , piece_id=JO42OKIDLNRW3X26C7LAFJJQUP3OTKMHIZTX7FRZAQV72J5JUI2Q, seg_pos={0 0}, upload=55991]: Uploading piece... Jul 26 21:56:25.000443117 (58h20m34.723006157s) [upload=55991]: Next returned nil segment Jul 26 21:56:25.000464077 (58h20m34.723027127s) [upload=55991]: Waiting for error group managing segments... Jul 26 21:56:25.286635891 (58h20m35.009198921s) [noise=true , address=45.91.116.7:4009 , piece_id=332GMURJ4CKAAJ5RYOKM42NDVJ3IQH7ZYIQ2WK3SVCTQVL2RSIAQ, seg_pos={0 0}, upload=55991]: Done uploading piece. err: <nil> Jul 26 21:56:25.296417787 (58h20m35.018980827s) [noise=true , address=45.91.116.5:4298 , piece_id=ZTBV2MODSZJK4XJF5B4AYOFW6BINA26XVRQPYWWI24JVQCXH562A, seg_pos={0 0}, upload=55991]: Done uploading piece. err: <nil> Jul 26 21:56:25.361692186 (58h20m35.084255226s) [noise=true , address=45.91.116.10:4083 , piece_id=LKTYNKD7SPUQPIC5CR4RWDRPVFS3TTIBBFARLAYYLGKFJHDZA24Q, seg_pos={0 0}, upload=55991]: Done uploading piece. err: <nil> Jul 26 21:56:25.368791419 (58h20m35.091354459s) [noise=true , address=45.91.116.6:4973 , piece_id=AVZ4VI6FJDY5443WB5GLQSVC4J74IKUKOLIEEZLVM4CC4X4P4B3Q, seg_pos={0 0}, upload=55991]: Done uploading piece. err: <nil> Jul 26 21:56:25.378604785 (58h20m35.101167825s) [noise=true , address=45.91.116.12:4227 , piece_id=FCQL5H52LUAZWOKIJWS6GUZANHD2E7YFJSPCABRORVQAVRXHX6ZA, seg_pos={0 0}, upload=55991]: Done uploading piece. err: <nil> Jul 26 21:56:25.432128342 (58h20m35.154691382s) [noise=true , address=45.91.116.11:4117 , piece_id=MDWEVNUPXCADXKDB24IWT4PDK4D62LOTZ6M4AFH76DYM4CWZRQZQ, seg_pos={0 0}, upload=55991]: Done uploading piece. err: <nil> Jul 26 21:56:25.440496382 (58h20m35.163059422s) [noise=true , address=45.91.116.4:4937 , piece_id=5Y5GFRONP23RJ6KUVUIVFUM37QMAUMKXIMMPW7YUHDCLHIERMASQ, seg_pos={0 0}, upload=55991]: Done uploading piece. err: <nil> Jul 26 21:56:25.442055686 (58h20m35.164618716s) [noise=true , address=45.91.116.6:4975 , piece_id=EDZ7SRCY7NNIWCHNMK5ENXKHJZTAVGLIU5UJX6EPNQBNBGZWDPOQ, seg_pos={0 0}, upload=55991]: Done uploading piece. err: <nil> Jul 26 21:56:25.456739370 (58h20m35.17930241s) [noise=true , address=45.91.116.11:4118 , piece_id=JO42OKIDLNRW3X26C7LAFJJQUP3OTKMHIZTX7FRZAQV72J5JUI2Q, seg_pos={0 0}, upload=55991]: Done uploading piece. err: <nil> Jul 26 21:56:31.170583990 (58h20m40.89314705s) [noise=true , address=77.90.31.15:4697 , piece_id=WJC4QDUAVNMJA7LKHVWRGDKXXYNYI53SJDD4WV4F4YZHKBCOCEYQ, seg_pos={0 0}, upload=55991]: Done uploading piece. err: <nil> Jul 26 21:56:31.200237537 (58h20m40.922800577s) [noise=true , address=77.90.31.10:4519 , piece_id=VBGTE6B73E7GIWTWRRBYDSK4FQCPQI2HQDJNXUA3LLNSNPADV6TQ, seg_pos={0 0}, upload=55991]: Done uploading piece. err: <nil> Jul 26 21:56:39.479107631 (58h20m49.201670671s) [noise=true , address=77.90.31.11:4555 , piece_id=M52GC3KWD7XO6FJC77ERY23TT3GYPPVSDAUTXICXWIDIXEFDADNQ, seg_pos={0 0}, upload=55991]: Done uploading piece. err: <nil> Jul 26 21:56:52.368471296 (58h21m2.091034336s) [noise=true , address=77.90.31.19:4843 , piece_id=56B4KRMMKSR7ZWS3RGT57PPOFRHVJBWXBC36HMQMOYKQWBMKIT2Q, seg_pos={0 0}, upload=55991]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 21:56:52.368604150 (58h21m2.09116719s) [noise=true , address=104.36.82.8:4969 , piece_id=VMB3FRYKZGXDAJFLR6GKAFUX7IM4V7BB55RHXMS7Q3ZG7MZVUUYQ, seg_pos={0 0}, upload=55991]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:43730->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:56:52.368656821 (58h21m2.091219861s) [noise=true , address=77.90.31.14:4661 , piece_id=UERZICXUIB2BY3YE77K3N7QUJ6S3Y3MNFFH23CN5DVCYSPLX7F7Q, seg_pos={0 0}, upload=55991]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:51296->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:51296->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51296->77.90.31.14:4661: use of closed network connection Jul 26 21:56:52.368664966 (58h21m2.091228006s) [noise=true , address=104.36.82.15:4258 , piece_id=YPGQT6YYV4E732W5DSB7NRCCZJCFEVC33NGO526N3Q5MYY3UWMMA, seg_pos={0 0}, upload=55991]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:52766->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:56:52.368697107 (58h21m2.091260147s) [seg_pos={0 0}, upload=55991]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 21:56:52.611389012 (58h21m2.333952052s) [upload=55991]: Done uploading segments... Jul 26 21:56:52.611414972 (58h21m2.333978002s) [upload=55991]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 17.249862918s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......X.................X..................................] duration: 1.925514056s FAIL (2025-07-26 21:57:34 UTC) orbiter warning: digest mismatch: after 0783c6a051cf3b535b7c61f8e6f0cfe751970c7121d9bce2182fd51e29ca8fc6 != before ee59801cb45aa958945ed73fcee20d9c3604e7f558c75f49df119aca36a29d88 Duration: 1.925556596s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................X..X.] duration: 11.391447329s FAIL (2025-07-26 21:57:18 UTC) orbiter warning: context deadline exceeded Duration: 11.391459682s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [........X.................X.................................] duration: 1.840197338s FAIL (2025-07-26 21:57:17 UTC) orbiter warning: digest mismatch: after 0783c6a051cf3b535b7c61f8e6f0cfe751970c7121d9bce2182fd51e29ca8fc6 != before ee59801cb45aa958945ed73fcee20d9c3604e7f558c75f49df119aca36a29d88 Duration: 1.84022456s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 28 out of 60 is failed (threshold=3) [................................XXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.239378273s FAIL (2025-07-26 21:57:10 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== .225:4449 , piece_id=N2NLHW5LJVWUNXJDPHTYIYOXVQVPXDKYBFTLUPC646TTT5FG45PA, seg_pos={0 0}, upload=55988]: Uploading piece... Jul 26 21:56:11.555052686 (58h20m21.277615726s) [noise=true , address=64.49.13.138:4482 , piece_id=NHESW2UDQICXITJZGOTQCJ7OXYAEX2HOFSAI262V6VIGVLGR2L2A, seg_pos={0 0}, upload=55988]: Uploading piece... Jul 26 21:56:11.555198305 (58h20m21.277761345s) [noise=true , address=77.90.31.15:4717 , piece_id=CYVZZZICWASTA6VGFHAW4256AK7TVLLE7BMFHKRJN3NWUA7QADQA, seg_pos={0 0}, upload=55988]: Uploading piece... Jul 26 21:56:11.555310108 (58h20m21.277873138s) [noise=true , address=104.36.82.9:4011 , piece_id=QOMCVNOG6UA6GCCZLONY7J6MBUUM3F4MJQDZ7PICGSAB5JIGQF6Q, seg_pos={0 0}, upload=55988]: Uploading piece... Jul 26 21:56:11.555442282 (58h20m21.278005322s) [noise=true , address=104.36.82.56:4264 , piece_id=WDMQNMRNRNAWONAVKO3ZGGFD4RJWVMZWSOHT7BEZFSEJZCRUX5PQ, seg_pos={0 0}, upload=55988]: Uploading piece... Jul 26 21:56:11.555535340 (58h20m21.27809838s) [noise=true , address=104.36.82.50:4016 , piece_id=FWHFLJUFD6DKU7MZAMUYGQWRDQD27UBJUD6BBTCS3NL7LIY7MRFQ, seg_pos={0 0}, upload=55988]: Uploading piece... Jul 26 21:56:11.555676801 (58h20m21.278239841s) [noise=true , address=104.36.82.47:4523 , piece_id=HRAKPYPLDRTV7UE3VHB6JKTW673KH3IMZYFADAFJZAPM6QAYXVHQ, seg_pos={0 0}, upload=55988]: Uploading piece... Jul 26 21:56:11.555766873 (58h20m21.278329903s) [noise=true , address=104.36.82.42:4825 , piece_id=NZWVIN4QSGAABLOYSYFKHQTAYHDX7775N2SDQ3VWWONSCFKC3VQA, seg_pos={0 0}, upload=55988]: Uploading piece... Jul 26 21:56:11.555923112 (58h20m21.278486152s) [noise=true , address=104.36.82.46:4496 , piece_id=FOCKGFFWKDEVXAFDUF5NAPDKRXXAH2UF263EMJOB2KCAVJWBQFHQ, seg_pos={0 0}, upload=55988]: Uploading piece... Jul 26 21:56:11.555994117 (58h20m21.278557157s) [noise=true , address=104.36.82.52:4119 , piece_id=PP7Q7RKPEL3W6FUXA2RPBRVZJB3L5A2ORK45K2XMIJFACMK5AFCA, seg_pos={0 0}, upload=55988]: Uploading piece... Jul 26 21:56:11.556153212 (58h20m21.278716242s) [noise=true , address=104.36.82.49:4011 , piece_id=AR4UJRBN4QVAWY5DTW2F2DA22NWIDV2Z567BGOYQWDVIARXVDFPQ, seg_pos={0 0}, upload=55988]: Uploading piece... Jul 26 21:56:11.556219058 (58h20m21.278782089s) [noise=true , address=77.90.31.21:4885 , piece_id=QYXCTYT22L6YTWW5SFMTD2IYJD54GM4QN7YKAAZS7NUJIEFOPT6A, seg_pos={0 0}, upload=55988]: Uploading piece... Jul 26 21:56:11.556459177 (58h20m21.279022207s) [noise=true , address=104.36.82.10:4089 , piece_id=YXPPFXITHEPT5PWBZ3665RSYJCDHLJ6VRAJIISZXCPEGICGSIIXQ, seg_pos={0 0}, upload=55988]: Uploading piece... Jul 26 21:56:12.645145597 (58h20m22.367708637s) [upload=55988]: Next returned nil segment Jul 26 21:56:12.645168511 (58h20m22.367731551s) [upload=55988]: Waiting for error group managing segments... Jul 26 21:56:12.979427010 (58h20m22.70199008s) [noise=true , address=79.127.205.235:4685 , piece_id=XYXNGQANLUA66AMM7U2VB24IZD44PWQZCR2LAZM2DL5EF3RL7RDQ, seg_pos={0 0}, upload=55988]: Done uploading piece. err: <nil> Jul 26 21:56:13.031295109 (58h20m22.753858149s) [noise=true , address=64.49.13.138:4482 , piece_id=NHESW2UDQICXITJZGOTQCJ7OXYAEX2HOFSAI262V6VIGVLGR2L2A, seg_pos={0 0}, upload=55988]: Done uploading piece. err: <nil> Jul 26 21:56:13.578270260 (58h20m23.30083329s) [noise=true , address=79.127.219.38:4018 , piece_id=ZVVK66XC7MWKI4Z3GWWWMZ6V5SLXPX7KJBZ2EUJGZ7WXSJP56IIA, seg_pos={0 0}, upload=55988]: Done uploading piece. err: <nil> Jul 26 21:56:13.633569864 (58h20m23.356132904s) [noise=true , address=79.127.219.51:4212 , piece_id=6F6EGEVPYM5ROMEFYIEQQW3RA5HLCWUZSYZ2HFPRU6NPUNV2GL6A, seg_pos={0 0}, upload=55988]: Done uploading piece. err: <nil> Jul 26 21:56:17.466240734 (58h20m27.188803774s) [noise=true , address=156.146.43.225:4449 , piece_id=N2NLHW5LJVWUNXJDPHTYIYOXVQVPXDKYBFTLUPC646TTT5FG45PA, seg_pos={0 0}, upload=55988]: Done uploading piece. err: <nil> Jul 26 21:56:25.756365818 (58h20m35.478928858s) [noise=true , address=77.90.31.21:4885 , piece_id=QYXCTYT22L6YTWW5SFMTD2IYJD54GM4QN7YKAAZS7NUJIEFOPT6A, seg_pos={0 0}, upload=55988]: Done uploading piece. err: <nil> Jul 26 21:56:26.890955014 (58h20m36.613518054s) [noise=true , address=77.90.31.15:4717 , piece_id=CYVZZZICWASTA6VGFHAW4256AK7TVLLE7BMFHKRJN3NWUA7QADQA, seg_pos={0 0}, upload=55988]: Done uploading piece. err: <nil> Jul 26 21:57:10.448487850 (58h21m20.17105089s) [noise=true , address=104.36.82.52:4119 , piece_id=PP7Q7RKPEL3W6FUXA2RPBRVZJB3L5A2ORK45K2XMIJFACMK5AFCA, seg_pos={0 0}, upload=55988]: Done uploading piece. err: ecclient: upload failed (node:12mTLjL6Y6VVjqztwVXYif4v7tyftjLafztLUibajksMHCFraFG, address:104.36.82.52:4119): protocol: write tcp 188.166.225.170:35586->104.36.82.52:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:57:10.448534067 (58h21m20.171097107s) [noise=true , address=104.36.82.10:4089 , piece_id=YXPPFXITHEPT5PWBZ3665RSYJCDHLJ6VRAJIISZXCPEGICGSIIXQ, seg_pos={0 0}, upload=55988]: Done uploading piece. err: ecclient: upload failed (node:12bFi2opJ6FejAMwSL2qbqWoCk24QzSjT6xQYE8BytLgVJikZMT, address:104.36.82.10:4089): protocol: write tcp 188.166.225.170:34662->104.36.82.10:4089: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:57:10.448574185 (58h21m20.171137225s) [noise=true , address=104.36.82.9:4011 , piece_id=QOMCVNOG6UA6GCCZLONY7J6MBUUM3F4MJQDZ7PICGSAB5JIGQF6Q, seg_pos={0 0}, upload=55988]: Done uploading piece. err: ecclient: upload failed (node:12J8J47emDh4p115m69EqUYroi9GgruC3y8gmv2xVULVJSvFYYP, address:104.36.82.9:4011): protocol: write tcp 188.166.225.170:52844->104.36.82.9:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:57:10.448613780 (58h21m20.17117682s) [noise=true , address=104.36.82.42:4825 , piece_id=NZWVIN4QSGAABLOYSYFKHQTAYHDX7775N2SDQ3VWWONSCFKC3VQA, seg_pos={0 0}, upload=55988]: Done uploading piece. err: ecclient: upload failed (node:12AYyhgZh7oAaGf4EXLSp6s1csFxCKfLuh6CJAPyzdC6pLQ51h, address:104.36.82.42:4825): protocol: write tcp 188.166.225.170:37948->104.36.82.42:4825: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:57:10.448659879 (58h21m20.171222909s) [noise=true , address=104.36.82.46:4496 , piece_id=FOCKGFFWKDEVXAFDUF5NAPDKRXXAH2UF263EMJOB2KCAVJWBQFHQ, seg_pos={0 0}, upload=55988]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: write tcp 188.166.225.170:53268->104.36.82.46:4496: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:57:10.448691479 (58h21m20.171254519s) [noise=true , address=104.36.82.47:4523 , piece_id=HRAKPYPLDRTV7UE3VHB6JKTW673KH3IMZYFADAFJZAPM6QAYXVHQ, seg_pos={0 0}, upload=55988]: Done uploading piece. err: ecclient: upload failed (node:12pADNfAT7ccugc1jKnTsk2xiw1aSEpuoKZSRosexzHowDmDCWV, address:104.36.82.47:4523): protocol: write tcp 188.166.225.170:41898->104.36.82.47:4523: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:57:10.448723871 (58h21m20.171286901s) [noise=true , address=104.36.82.54:4160 , piece_id=RVIPFCA3E3XB44PGW5PQ4DPANUIRSXG6E5YLWQS4XYR5SOQN53FQ, seg_pos={0 0}, upload=55988]: Done uploading piece. err: ecclient: upload failed (node:12qRsRx7vmb3S9m5J7gceK44gnuuYYMyorSAyxcpjpPeRJJFZ84, address:104.36.82.54:4160): protocol: write tcp 188.166.225.170:42732->104.36.82.54:4160: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:57:10.448753848 (58h21m20.171316888s) [noise=true , address=104.36.82.53:4146 , piece_id=N377UMONYLWYHCSOML3I4JOLNLYUO3OLXZVFMYIASSDH657MVE7Q, seg_pos={0 0}, upload=55988]: Done uploading piece. err: ecclient: upload failed (node:1ecwZFchyb5HmyKwRFfLDaVGck3xDWmgrVNYadHoTLWY5Z2Qs1, address:104.36.82.53:4146): protocol: write tcp 188.166.225.170:59010->104.36.82.53:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:57:10.448785630 (58h21m20.17134866s) [noise=true , address=104.36.82.56:4264 , piece_id=WDMQNMRNRNAWONAVKO3ZGGFD4RJWVMZWSOHT7BEZFSEJZCRUX5PQ, seg_pos={0 0}, upload=55988]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: write tcp 188.166.225.170:38504->104.36.82.56:4264: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:57:10.448814906 (58h21m20.171377946s) [noise=true , address=104.36.82.49:4011 , piece_id=AR4UJRBN4QVAWY5DTW2F2DA22NWIDV2Z567BGOYQWDVIARXVDFPQ, seg_pos={0 0}, upload=55988]: Done uploading piece. err: ecclient: upload failed (node:1DV1WUL3dGBjH23a6xtetmqmLE9fER1FQf53gmYFxZiP25ra8N, address:104.36.82.49:4011): protocol: write tcp 188.166.225.170:35044->104.36.82.49:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:57:10.448842979 (58h21m20.171406009s) [noise=true , address=104.36.82.50:4016 , piece_id=FWHFLJUFD6DKU7MZAMUYGQWRDQD27UBJUD6BBTCS3NL7LIY7MRFQ, seg_pos={0 0}, upload=55988]: Done uploading piece. err: ecclient: upload failed (node:1jC98jEWn6K8jFHPA9knJeHAkuNEmcS7ThxUvbySDfbq6FmKVR, address:104.36.82.50:4016): protocol: write tcp 188.166.225.170:42380->104.36.82.50:4016: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:57:10.448856715 (58h21m20.171419745s) [seg_pos={0 0}, upload=55988]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 26 21:57:10.685390968 (58h21m20.407954008s) [upload=55988]: Done uploading segments... Jul 26 21:57:10.685415785 (58h21m20.407978825s) [upload=55988]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.239409092s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................................X..X] duration: 30.001294623s FAIL (2025-07-26 21:57:07 UTC) orbiter warning: context deadline exceeded Duration: 30.001321124s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 16 out of 60 is failed (threshold=3) [...............................XX.........XX.XX.XXXX.XX.XXXX] duration: 30.231146575s FAIL (2025-07-26 21:56:58 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.231166313s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [........................X....X......XX....XX.XX.XXX.X.XX.XXX] duration: 30.244775005s FAIL (2025-07-26 21:56:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== d Jul 26 21:56:21.880426452 (58h20m31.602989492s) [noise=true , address=104.36.82.8:4969 , piece_id=7JFRXSBU7C3FVWXI4A6XV7WCDZQAV7VJPC4UJ2O7NQFVT6D4IM6A, seg_pos={0 0}, upload=55985]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: context deadline exceeded; context deadline exceeded Jul 26 21:56:21.880444877 (58h20m31.603007917s) [seg_pos={0 0}, upload=55985]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 21:56:22.111166668 (58h20m31.833729708s) [upload=55985]: Done uploading segments... Jul 26 21:56:22.111209851 (58h20m31.833772891s) [upload=55985]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 26 21:56:23.019512381 (58h20m32.742075421s) [upload=55991]: Starting upload Jul 26 21:56:23.019569490 (58h20m32.74213253s) [upload=55991]: Done starting upload Jul 26 21:56:23.019678088 (58h20m32.742241128s) [upload=55991]: Uploading segments... Jul 26 21:56:23.019837513 (58h20m32.742400573s) [upload=55991]: Got next segment. Inline: false Jul 26 21:56:23.019881758 (58h20m32.742444838s) [upload=55991]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 21:56:23.019890524 (58h20m32.742453574s) [upload=55991]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 21:56:23.255552398 (58h20m32.978115438s) [seg_pos={0 0}, upload=55991]: Begin upload segment... Jul 26 21:56:23.255606421 (58h20m32.978169451s) [seg_pos={0 0}, upload=55991]: Done begin upload segment. Jul 26 21:56:23.255629285 (58h20m32.978192325s) [noise=true , address=45.91.116.6:4975 , piece_id=EDZ7SRCY7NNIWCHNMK5ENXKHJZTAVGLIU5UJX6EPNQBNBGZWDPOQ, seg_pos={0 0}, upload=55991]: Uploading piece... Jul 26 21:56:23.255656206 (58h20m32.978219236s) [noise=true , address=45.91.116.12:4227 , piece_id=FCQL5H52LUAZWOKIJWS6GUZANHD2E7YFJSPCABRORVQAVRXHX6ZA, seg_pos={0 0}, upload=55991]: Uploading piece... Jul 26 21:56:23.256097681 (58h20m32.978660721s) [noise=true , address=45.91.116.5:4298 , piece_id=ZTBV2MODSZJK4XJF5B4AYOFW6BINA26XVRQPYWWI24JVQCXH562A, seg_pos={0 0}, upload=55991]: Uploading piece... Jul 26 21:56:23.256426109 (58h20m32.978989149s) [noise=true , address=104.36.82.15:4258 , piece_id=YPGQT6YYV4E732W5DSB7NRCCZJCFEVC33NGO526N3Q5MYY3UWMMA, seg_pos={0 0}, upload=55991]: Uploading piece... Jul 26 21:56:23.256710143 (58h20m32.979273183s) [noise=true , address=45.91.116.6:4973 , piece_id=AVZ4VI6FJDY5443WB5GLQSVC4J74IKUKOLIEEZLVM4CC4X4P4B3Q, seg_pos={0 0}, upload=55991]: Uploading piece... Jul 26 21:56:23.256774927 (58h20m32.979337967s) [noise=true , address=45.91.116.11:4117 , piece_id=MDWEVNUPXCADXKDB24IWT4PDK4D62LOTZ6M4AFH76DYM4CWZRQZQ, seg_pos={0 0}, upload=55991]: Uploading piece... Jul 26 21:56:23.257064120 (58h20m32.97962716s) [noise=true , address=104.36.82.8:4969 , piece_id=VMB3FRYKZGXDAJFLR6GKAFUX7IM4V7BB55RHXMS7Q3ZG7MZVUUYQ, seg_pos={0 0}, upload=55991]: Uploading piece... Jul 26 21:56:23.257312285 (58h20m32.979875315s) [noise=true , address=77.90.31.15:4697 , piece_id=WJC4QDUAVNMJA7LKHVWRGDKXXYNYI53SJDD4WV4F4YZHKBCOCEYQ, seg_pos={0 0}, upload=55991]: Uploading piece... Jul 26 21:56:23.257611768 (58h20m32.980174808s) [noise=true , address=77.90.31.11:4555 , piece_id=M52GC3KWD7XO6FJC77ERY23TT3GYPPVSDAUTXICXWIDIXEFDADNQ, seg_pos={0 0}, upload=55991]: Uploading piece... Jul 26 21:56:23.257909758 (58h20m32.980472798s) [noise=true , address=77.90.31.10:4519 , piece_id=VBGTE6B73E7GIWTWRRBYDSK4FQCPQI2HQDJNXUA3LLNSNPADV6TQ, seg_pos={0 0}, upload=55991]: Uploading piece... Jul 26 21:56:23.257954583 (58h20m32.980517623s) [noise=true , address=45.91.116.4:4937 , piece_id=5Y5GFRONP23RJ6KUVUIVFUM37QMAUMKXIMMPW7YUHDCLHIERMASQ, seg_pos={0 0}, upload=55991]: Uploading piece... Jul 26 21:56:23.258203630 (58h20m32.98076667s) [noise=true , address=45.91.116.7:4009 , piece_id=332GMURJ4CKAAJ5RYOKM42NDVJ3IQH7ZYIQ2WK3SVCTQVL2RSIAQ, seg_pos={0 0}, upload=55991]: Uploading piece... Jul 26 21:56:23.258431867 (58h20m32.980994907s) [noise=true , address=77.90.31.14:4661 , piece_id=UERZICXUIB2BY3YE77K3N7QUJ6S3Y3MNFFH23CN5DVCYSPLX7F7Q, seg_pos={0 0}, upload=55991]: Uploading piece... Jul 26 21:56:23.258523622 (58h20m32.981086662s) [noise=true , address=45.91.116.10:4083 , piece_id=LKTYNKD7SPUQPIC5CR4RWDRPVFS3TTIBBFARLAYYLGKFJHDZA24Q, seg_pos={0 0}, upload=55991]: Uploading piece... Jul 26 21:56:23.258820911 (58h20m32.981383951s) [noise=true , address=77.90.31.19:4843 , piece_id=56B4KRMMKSR7ZWS3RGT57PPOFRHVJBWXBC36HMQMOYKQWBMKIT2Q, seg_pos={0 0}, upload=55991]: Uploading piece... Jul 26 21:56:23.258976820 (58h20m32.98153985s) [noise=true , address=45.91.116.11:4118 , piece_id=JO42OKIDLNRW3X26C7LAFJJQUP3OTKMHIZTX7FRZAQV72J5JUI2Q, seg_pos={0 0}, upload=55991]: Uploading piece... Jul 26 21:56:25.000443117 (58h20m34.723006157s) [upload=55991]: Next returned nil segment Jul 26 21:56:25.000464077 (58h20m34.723027127s) [upload=55991]: Waiting for error group managing segments... Jul 26 21:56:25.286635891 (58h20m35.009198921s) [noise=true , address=45.91.116.7:4009 , piece_id=332GMURJ4CKAAJ5RYOKM42NDVJ3IQH7ZYIQ2WK3SVCTQVL2RSIAQ, seg_pos={0 0}, upload=55991]: Done uploading piece. err: <nil> Jul 26 21:56:25.296417787 (58h20m35.018980827s) [noise=true , address=45.91.116.5:4298 , piece_id=ZTBV2MODSZJK4XJF5B4AYOFW6BINA26XVRQPYWWI24JVQCXH562A, seg_pos={0 0}, upload=55991]: Done uploading piece. err: <nil> Jul 26 21:56:25.361692186 (58h20m35.084255226s) [noise=true , address=45.91.116.10:4083 , piece_id=LKTYNKD7SPUQPIC5CR4RWDRPVFS3TTIBBFARLAYYLGKFJHDZA24Q, seg_pos={0 0}, upload=55991]: Done uploading piece. err: <nil> Jul 26 21:56:25.368791419 (58h20m35.091354459s) [noise=true , address=45.91.116.6:4973 , piece_id=AVZ4VI6FJDY5443WB5GLQSVC4J74IKUKOLIEEZLVM4CC4X4P4B3Q, seg_pos={0 0}, upload=55991]: Done uploading piece. err: <nil> Jul 26 21:56:25.378604785 (58h20m35.101167825s) [noise=true , address=45.91.116.12:4227 , piece_id=FCQL5H52LUAZWOKIJWS6GUZANHD2E7YFJSPCABRORVQAVRXHX6ZA, seg_pos={0 0}, upload=55991]: Done uploading piece. err: <nil> Jul 26 21:56:25.432128342 (58h20m35.154691382s) [noise=true , address=45.91.116.11:4117 , piece_id=MDWEVNUPXCADXKDB24IWT4PDK4D62LOTZ6M4AFH76DYM4CWZRQZQ, seg_pos={0 0}, upload=55991]: Done uploading piece. err: <nil> Jul 26 21:56:25.440496382 (58h20m35.163059422s) [noise=true , address=45.91.116.4:4937 , piece_id=5Y5GFRONP23RJ6KUVUIVFUM37QMAUMKXIMMPW7YUHDCLHIERMASQ, seg_pos={0 0}, upload=55991]: Done uploading piece. err: <nil> Jul 26 21:56:25.442055686 (58h20m35.164618716s) [noise=true , address=45.91.116.6:4975 , piece_id=EDZ7SRCY7NNIWCHNMK5ENXKHJZTAVGLIU5UJX6EPNQBNBGZWDPOQ, seg_pos={0 0}, upload=55991]: Done uploading piece. err: <nil> Jul 26 21:56:25.456739370 (58h20m35.17930241s) [noise=true , address=45.91.116.11:4118 , piece_id=JO42OKIDLNRW3X26C7LAFJJQUP3OTKMHIZTX7FRZAQV72J5JUI2Q, seg_pos={0 0}, upload=55991]: Done uploading piece. err: <nil> Jul 26 21:56:31.170583990 (58h20m40.89314705s) [noise=true , address=77.90.31.15:4697 , piece_id=WJC4QDUAVNMJA7LKHVWRGDKXXYNYI53SJDD4WV4F4YZHKBCOCEYQ, seg_pos={0 0}, upload=55991]: Done uploading piece. err: <nil> Jul 26 21:56:31.200237537 (58h20m40.922800577s) [noise=true , address=77.90.31.10:4519 , piece_id=VBGTE6B73E7GIWTWRRBYDSK4FQCPQI2HQDJNXUA3LLNSNPADV6TQ, seg_pos={0 0}, upload=55991]: Done uploading piece. err: <nil> Jul 26 21:56:39.479107631 (58h20m49.201670671s) [noise=true , address=77.90.31.11:4555 , piece_id=M52GC3KWD7XO6FJC77ERY23TT3GYPPVSDAUTXICXWIDIXEFDADNQ, seg_pos={0 0}, upload=55991]: Done uploading piece. err: <nil> Jul 26 21:56:52.368471296 (58h21m2.091034336s) [noise=true , address=77.90.31.19:4843 , piece_id=56B4KRMMKSR7ZWS3RGT57PPOFRHVJBWXBC36HMQMOYKQWBMKIT2Q, seg_pos={0 0}, upload=55991]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 21:56:52.368604150 (58h21m2.09116719s) [noise=true , address=104.36.82.8:4969 , piece_id=VMB3FRYKZGXDAJFLR6GKAFUX7IM4V7BB55RHXMS7Q3ZG7MZVUUYQ, seg_pos={0 0}, upload=55991]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: write tcp 188.166.225.170:43730->104.36.82.8:4969: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:56:52.368656821 (58h21m2.091219861s) [noise=true , address=77.90.31.14:4661 , piece_id=UERZICXUIB2BY3YE77K3N7QUJ6S3Y3MNFFH23CN5DVCYSPLX7F7Q, seg_pos={0 0}, upload=55991]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:51296->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:51296->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51296->77.90.31.14:4661: use of closed network connection Jul 26 21:56:52.368664966 (58h21m2.091228006s) [noise=true , address=104.36.82.15:4258 , piece_id=YPGQT6YYV4E732W5DSB7NRCCZJCFEVC33NGO526N3Q5MYY3UWMMA, seg_pos={0 0}, upload=55991]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: write tcp 188.166.225.170:52766->104.36.82.15:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:56:52.368697107 (58h21m2.091260147s) [seg_pos={0 0}, upload=55991]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 21:56:52.611389012 (58h21m2.333952052s) [upload=55991]: Done uploading segments... Jul 26 21:56:52.611414972 (58h21m2.333978002s) [upload=55991]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.244787309s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........X.................X................................] duration: 1.854308798s FAIL (2025-07-26 21:56:35 UTC) orbiter warning: digest mismatch: after 0783c6a051cf3b535b7c61f8e6f0cfe751970c7121d9bce2182fd51e29ca8fc6 != before ee59801cb45aa958945ed73fcee20d9c3604e7f558c75f49df119aca36a29d88 Duration: 1.854337182s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.........................X....X......XX....XX.XX.XXX.X.XX.XX] duration: 30.232578015s FAIL (2025-07-26 21:56:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== s=104.36.82.16:4293 , piece_id=U3OAEB3PUGWJUWHOVJ4VHNJHIBUQ6UDVX2UUEEFV6BZ6RCF67QSQ, seg_pos={0 0}, upload=55975]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE): protocol: rpc: dial tcp 104.36.82.16:4293: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 21:55:47.785831891 (58h19m57.508394931s) [seg_pos={0 0}, upload=55975]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 21:55:47.785917705 (58h19m57.508480745s) [upload=55975]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 21:55:47.785922183 (58h19m57.508485223s) [upload=55975]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 21:55:48.054234566 (58h19m57.776797606s) [upload=55975]: Done uploading segments... Jul 26 21:55:48.054258502 (58h19m57.776821542s) [upload=55975]: Upload finished. err: metaclient: context deadline exceeded Jul 26 21:55:52.529081395 (58h20m2.251644435s) [upload=55985]: Starting upload Jul 26 21:55:52.529137583 (58h20m2.251700623s) [upload=55985]: Done starting upload Jul 26 21:55:52.529218147 (58h20m2.251781187s) [upload=55985]: Uploading segments... Jul 26 21:55:52.529310794 (58h20m2.251873834s) [upload=55985]: Got next segment. Inline: false Jul 26 21:55:52.529348927 (58h20m2.251911967s) [upload=55985]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 21:55:52.529354558 (58h20m2.251917598s) [upload=55985]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 21:55:52.769757122 (58h20m2.492320172s) [seg_pos={0 0}, upload=55985]: Begin upload segment... Jul 26 21:55:52.769839429 (58h20m2.492402469s) [seg_pos={0 0}, upload=55985]: Done begin upload segment. Jul 26 21:55:52.769866441 (58h20m2.492429481s) [noise=true , address=45.91.116.13:4153 , piece_id=NJUGVVCQQR6WYS43GTTVR5SHWM2LXCBGWBLIZ7JNB66ZYCD5HW7Q, seg_pos={0 0}, upload=55985]: Uploading piece... Jul 26 21:55:52.769894785 (58h20m2.492457825s) [noise=true , address=45.91.116.7:4011 , piece_id=EDQDA5W6Z2XH36CI7MUJVCYBNS2OJD64M5EROHFDNKP2KSOYB7WA, seg_pos={0 0}, upload=55985]: Uploading piece... Jul 26 21:55:52.770345247 (58h20m2.492908277s) [noise=true , address=45.91.116.11:4117 , piece_id=AQKX563BKR7TOPA2L4AI5MVKHMSPSG7GHN22KKWJN7F36TE2J2VA, seg_pos={0 0}, upload=55985]: Uploading piece... Jul 26 21:55:52.770362901 (58h20m2.492925941s) [noise=true , address=77.90.31.17:4772 , piece_id=HZCW32CNRTWKIP74KFIEHRVNJNHALNM3645PKOMCUPKPCHGJQI2Q, seg_pos={0 0}, upload=55985]: Uploading piece... Jul 26 21:55:52.770698643 (58h20m2.493261683s) [noise=true , address=45.91.116.5:4297 , piece_id=6DUNVWDNV3FN23KA4R6J37JSQDTQXH42V4O5I7RHO67EWUVR3JAQ, seg_pos={0 0}, upload=55985]: Uploading piece... Jul 26 21:55:52.770712830 (58h20m2.49327587s) [noise=true , address=104.36.82.8:4969 , piece_id=7JFRXSBU7C3FVWXI4A6XV7WCDZQAV7VJPC4UJ2O7NQFVT6D4IM6A, seg_pos={0 0}, upload=55985]: Uploading piece... Jul 26 21:55:52.771049875 (58h20m2.493612915s) [noise=true , address=77.90.31.12:4592 , piece_id=26FDTWOCGERHUP3X6D4OIEW3KHWT3PLPF7LEGC43EMNTHMEOUWXA, seg_pos={0 0}, upload=55985]: Uploading piece... Jul 26 21:55:52.771098077 (58h20m2.493661117s) [noise=true , address=104.36.82.5:4837 , piece_id=RSYHYJAUO3VU76UBW3EOMS42B6QXWRMB6MLMGNVLDNYGKBUOZNKA, seg_pos={0 0}, upload=55985]: Uploading piece... Jul 26 21:55:52.771381810 (58h20m2.49394485s) [noise=true , address=45.91.116.6:4973 , piece_id=R22ZN6I5BF7CLT2T6HJ5GSPLDNLQMNMP5NXXY235PTP4FTB3MO4Q, seg_pos={0 0}, upload=55985]: Uploading piece... Jul 26 21:55:52.771703476 (58h20m2.494266516s) [noise=true , address=45.91.116.14:4189 , piece_id=3OWWC3EOSHKII4QTXIHHJWM7Z4SAI7UU6BE2R3KLUS5YSRRER3KA, seg_pos={0 0}, upload=55985]: Uploading piece... Jul 26 21:55:52.772006636 (58h20m2.494569686s) [noise=true , address=77.90.31.16:4734 , piece_id=NJ4GZJA3FQA6SHNSXSDEDA6CUAY6MWHYUIKYZOFUMSF74Z3RJHDQ, seg_pos={0 0}, upload=55985]: Uploading piece... Jul 26 21:55:52.772368027 (58h20m2.494931077s) [noise=true , address=45.91.116.10:4083 , piece_id=Z43FCOG5IY4FERX3JBSNXO6NXTWZBST6FM7IWVAL6SBK6NA2L6BQ, seg_pos={0 0}, upload=55985]: Uploading piece... Jul 26 21:55:52.772676147 (58h20m2.495239187s) [noise=true , address=45.91.116.5:4298 , piece_id=YAPKIBX5EPGMA5WFZHJIFRODMXP7V5SIB4POCZLRTM4KSFXPKCFA, seg_pos={0 0}, upload=55985]: Uploading piece... Jul 26 21:55:52.772958727 (58h20m2.495521767s) [noise=true , address=45.91.116.13:4155 , piece_id=4TF2FGB322EJ67I5KA2PIT6OH5WJ2KENBQCOXWDEIZ6LNJWUCIXQ, seg_pos={0 0}, upload=55985]: Uploading piece... Jul 26 21:55:52.773256647 (58h20m2.495819687s) [noise=true , address=45.91.116.4:4937 , piece_id=YM5OZ2ALZJ7YWFQNC3PBDIOUTUR77TISGCLM4LWPEMQEHS4PDPYQ, seg_pos={0 0}, upload=55985]: Uploading piece... Jul 26 21:55:52.773549798 (58h20m2.496112839s) [noise=true , address=77.90.31.13:4626 , piece_id=GRBU3JG64WLQ2LFUTLZUJXPZLBNIYVQAM5D4RC5UM5QNTTQEZM4Q, seg_pos={0 0}, upload=55985]: Uploading piece... Jul 26 21:55:54.518424325 (58h20m4.240987365s) [upload=55985]: Next returned nil segment Jul 26 21:55:54.518445426 (58h20m4.241008466s) [upload=55985]: Waiting for error group managing segments... Jul 26 21:55:54.794791280 (58h20m4.51735432s) [noise=true , address=45.91.116.6:4973 , piece_id=R22ZN6I5BF7CLT2T6HJ5GSPLDNLQMNMP5NXXY235PTP4FTB3MO4Q, seg_pos={0 0}, upload=55985]: Done uploading piece. err: <nil> Jul 26 21:55:54.868775053 (58h20m4.591338093s) [noise=true , address=45.91.116.14:4189 , piece_id=3OWWC3EOSHKII4QTXIHHJWM7Z4SAI7UU6BE2R3KLUS5YSRRER3KA, seg_pos={0 0}, upload=55985]: Done uploading piece. err: <nil> Jul 26 21:55:54.875865099 (58h20m4.598428169s) [noise=true , address=45.91.116.13:4155 , piece_id=4TF2FGB322EJ67I5KA2PIT6OH5WJ2KENBQCOXWDEIZ6LNJWUCIXQ, seg_pos={0 0}, upload=55985]: Done uploading piece. err: <nil> Jul 26 21:55:54.876955796 (58h20m4.599518876s) [noise=true , address=45.91.116.7:4011 , piece_id=EDQDA5W6Z2XH36CI7MUJVCYBNS2OJD64M5EROHFDNKP2KSOYB7WA, seg_pos={0 0}, upload=55985]: Done uploading piece. err: <nil> Jul 26 21:55:54.881333422 (58h20m4.603896482s) [noise=true , address=45.91.116.5:4297 , piece_id=6DUNVWDNV3FN23KA4R6J37JSQDTQXH42V4O5I7RHO67EWUVR3JAQ, seg_pos={0 0}, upload=55985]: Done uploading piece. err: <nil> Jul 26 21:55:54.885471599 (58h20m4.608034639s) [noise=true , address=45.91.116.13:4153 , piece_id=NJUGVVCQQR6WYS43GTTVR5SHWM2LXCBGWBLIZ7JNB66ZYCD5HW7Q, seg_pos={0 0}, upload=55985]: Done uploading piece. err: <nil> Jul 26 21:55:54.885710205 (58h20m4.608273246s) [noise=true , address=45.91.116.11:4117 , piece_id=AQKX563BKR7TOPA2L4AI5MVKHMSPSG7GHN22KKWJN7F36TE2J2VA, seg_pos={0 0}, upload=55985]: Done uploading piece. err: <nil> Jul 26 21:55:54.894950986 (58h20m4.617514026s) [noise=true , address=45.91.116.4:4937 , piece_id=YM5OZ2ALZJ7YWFQNC3PBDIOUTUR77TISGCLM4LWPEMQEHS4PDPYQ, seg_pos={0 0}, upload=55985]: Done uploading piece. err: <nil> Jul 26 21:55:54.895238006 (58h20m4.617801046s) [noise=true , address=45.91.116.10:4083 , piece_id=Z43FCOG5IY4FERX3JBSNXO6NXTWZBST6FM7IWVAL6SBK6NA2L6BQ, seg_pos={0 0}, upload=55985]: Done uploading piece. err: <nil> Jul 26 21:55:54.994135206 (58h20m4.716698246s) [noise=true , address=45.91.116.5:4298 , piece_id=YAPKIBX5EPGMA5WFZHJIFRODMXP7V5SIB4POCZLRTM4KSFXPKCFA, seg_pos={0 0}, upload=55985]: Done uploading piece. err: <nil> Jul 26 21:56:05.549045663 (58h20m15.271608693s) [noise=true , address=77.90.31.13:4626 , piece_id=GRBU3JG64WLQ2LFUTLZUJXPZLBNIYVQAM5D4RC5UM5QNTTQEZM4Q, seg_pos={0 0}, upload=55985]: Done uploading piece. err: <nil> Jul 26 21:56:11.196569659 (58h20m20.919132699s) [noise=true , address=77.90.31.17:4772 , piece_id=HZCW32CNRTWKIP74KFIEHRVNJNHALNM3645PKOMCUPKPCHGJQI2Q, seg_pos={0 0}, upload=55985]: Done uploading piece. err: <nil> Jul 26 21:56:21.880002821 (58h20m31.602565852s) [noise=true , address=77.90.31.12:4592 , piece_id=26FDTWOCGERHUP3X6D4OIEW3KHWT3PLPF7LEGC43EMNTHMEOUWXA, seg_pos={0 0}, upload=55985]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 21:56:21.880200069 (58h20m31.602763109s) [noise=true , address=77.90.31.16:4734 , piece_id=NJ4GZJA3FQA6SHNSXSDEDA6CUAY6MWHYUIKYZOFUMSF74Z3RJHDQ, seg_pos={0 0}, upload=55985]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:58064->77.90.31.16:4734: use of closed network connection; write tcp 188.166.225.170:58064->77.90.31.16:4734: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58064->77.90.31.16:4734: use of closed network connection Jul 26 21:56:21.880270324 (58h20m31.602833354s) [noise=true , address=104.36.82.5:4837 , piece_id=RSYHYJAUO3VU76UBW3EOMS42B6QXWRMB6MLMGNVLDNYGKBUOZNKA, seg_pos={0 0}, upload=55985]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: context deadline exceeded; context deadline exceeded Jul 26 21:56:21.880426452 (58h20m31.602989492s) [noise=true , address=104.36.82.8:4969 , piece_id=7JFRXSBU7C3FVWXI4A6XV7WCDZQAV7VJPC4UJ2O7NQFVT6D4IM6A, seg_pos={0 0}, upload=55985]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: context deadline exceeded; context deadline exceeded Jul 26 21:56:21.880444877 (58h20m31.603007917s) [seg_pos={0 0}, upload=55985]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 21:56:22.111166668 (58h20m31.833729708s) [upload=55985]: Done uploading segments... Jul 26 21:56:22.111209851 (58h20m31.833772891s) [upload=55985]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.232608623s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 9.679069738s FAIL (2025-07-26 21:56:22 UTC) orbiter warning: context deadline exceeded Duration: 9.679126978s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)