Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 10.382261442s FAIL (2025-07-16 15:32:17 UTC) orbiter warning: context deadline exceeded Duration: 10.382286361s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 23.926559382s FAIL (2025-07-16 15:32:04 UTC) orbiter warning: context deadline exceeded Duration: 23.926575494s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................................................XXXX....] duration: 5.797775424s FAIL (2025-07-16 15:32:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.797802927s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...............................................X..XX........] duration: 6.550837212s FAIL (2025-07-16 15:31:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 6.55087215s core uplink us-select-1 us1-placement-20
checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 1.088415116s remote: "tcp:5.161.37.179:443" response headers: Content-Length=0 Date=Wed, 16 Jul 2025 15:03:45 GMT Access-Control-Allow-Origin=* target: https://www.storj-ipfs.com/api/v0/add FAIL (2025-07-16 15:31:39 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "" Duration: 1.088434375s edge ipfs us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001538104s FAIL (2025-07-16 15:31:26 UTC) orbiter warning: context deadline exceeded Duration: 30.001558853s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [................................................X..XX.......] duration: 5.820167485s FAIL (2025-07-16 15:31:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.820205218s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................................................XXXX...] duration: 6.160766881s FAIL (2025-07-16 15:30:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.160807989s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.................................................X..XX......] duration: 5.97593848s FAIL (2025-07-16 15:30:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.975970702s core uplink us-select-1 us1-placement-20
checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 916.577324ms remote: "tcp:5.161.37.179:443" response headers: Content-Length=0 Date=Wed, 16 Jul 2025 15:03:45 GMT Access-Control-Allow-Origin=* target: https://www.storj-ipfs.com/api/v0/add FAIL (2025-07-16 15:30:24 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "" Duration: 916.599204ms edge ipfs us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................................................XXXX..] duration: 6.023457766s FAIL (2025-07-16 15:30:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.023519765s core uplink us-select-1
checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 1.043177753s remote: "tcp:5.161.37.179:443" response headers: Content-Length=0 Date=Wed, 16 Jul 2025 15:03:45 GMT Access-Control-Allow-Origin=* target: https://www.storj-ipfs.com/api/v0/add FAIL (2025-07-16 15:29:33 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "" Duration: 1.04319737s edge ipfs us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..................................................X..XX.....] duration: 5.779401844s FAIL (2025-07-16 15:29:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.779423294s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...................................................X..XX....] duration: 26.720513923s FAIL (2025-07-16 15:29:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 26.720538019s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................................................XXXX.] duration: 5.864795756s FAIL (2025-07-16 15:29:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.864820745s core uplink us-select-1
checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 899.558018ms remote: "tcp:5.161.37.179:443" response headers: Content-Length=0 Date=Wed, 16 Jul 2025 15:03:45 GMT Access-Control-Allow-Origin=* target: https://www.storj-ipfs.com/api/v0/add FAIL (2025-07-16 15:29:06 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "" Duration: 899.57452ms edge ipfs us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [....................................................X..XX...] duration: 25.631575382s FAIL (2025-07-16 15:28:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 25.631605719s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................................................XXXX] duration: 30.23953394s FAIL (2025-07-16 15:28:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.239557275s core uplink us-select-1
checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 955.335852ms remote: "tcp:5.161.37.179:443" response headers: Content-Length=0 Date=Wed, 16 Jul 2025 15:03:45 GMT Access-Control-Allow-Origin=* target: https://www.storj-ipfs.com/api/v0/add FAIL (2025-07-16 15:28:02 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "" Duration: 955.367073ms edge ipfs us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.....................................................X..XX..] duration: 5.773031739s FAIL (2025-07-16 15:27:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.773055505s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................................XXX] duration: 30.23365843s FAIL (2025-07-16 15:27:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 38054618s) [seg_pos={0 0}, upload=446444]: Begin upload segment... Jul 16 15:26:42.167457117 (558h16m16.938098452s) [seg_pos={0 0}, upload=446444]: Done begin upload segment. Jul 16 15:26:42.167478057 (558h16m16.938119382s) [noise=true , address=77.90.31.11:4554 , piece_id=CAV37AKARHLV36PGYWCJ6LQML6RNFM6QYMC232PZAE2TNCZJNGWQ, seg_pos={0 0}, upload=446444]: Uploading piece... Jul 16 15:26:42.167558421 (558h16m16.938199756s) [noise=true , address=77.90.31.13:4627 , piece_id=VFJIJMDVJA62SW55JUGRYB67DP3WQ2OFB5ZB6BLHZFE3NCYERISA, seg_pos={0 0}, upload=446444]: Uploading piece... Jul 16 15:26:42.167830922 (558h16m16.938472258s) [noise=true , address=104.36.82.13:4185 , piece_id=HYKG4PUJRBHFVQBFPSCJU3QBFYZEBWURM4BFQUZIEHIKZMCJQBVQ, seg_pos={0 0}, upload=446444]: Uploading piece... Jul 16 15:26:42.167990958 (558h16m16.938632293s) [noise=true , address=104.36.82.16:4295 , piece_id=2SGFTI7BERIIRNOSWMHN4TIFBNIEQTHD4ZM2YAGENOI22LC76W4A, seg_pos={0 0}, upload=446444]: Uploading piece... Jul 16 15:26:42.168107341 (558h16m16.938748676s) [noise=true , address=45.91.116.7:4011 , piece_id=4LTNQL5LI75YNKEID25D2AHJDKIHQNBOMQ7GMGUCHNP3YIEO5ADQ, seg_pos={0 0}, upload=446444]: Uploading piece... Jul 16 15:26:42.168260113 (558h16m16.938901438s) [noise=true , address=45.91.116.10:4081 , piece_id=B5V5HZKLYKGUNMYXTOURMR66NSXNP3ZOI5ZZMMRCEGXFOXR6NMBQ, seg_pos={0 0}, upload=446444]: Uploading piece... Jul 16 15:26:42.168368130 (558h16m16.939009455s) [noise=true , address=45.91.116.10:4082 , piece_id=LP3SERHWRQOC2JIESAM5JBH6NY4Q3N5WWTPFN4UHJR2PJ5O3OIRA, seg_pos={0 0}, upload=446444]: Uploading piece... Jul 16 15:26:42.168532033 (558h16m16.939173358s) [noise=true , address=104.36.82.6:4874 , piece_id=H4BZTYSK5C3M2RKCGJFPAUCBMFMUD2TWRXX6NZ7JEGJMWTF7HD7A, seg_pos={0 0}, upload=446444]: Uploading piece... Jul 16 15:26:42.168675568 (558h16m16.939316903s) [noise=true , address=45.91.116.9:4261 , piece_id=7PYHMSQIXZOSBOFYVRTCDVWD23HU2LQKPCLN7XSRJWD3KRK64IQQ, seg_pos={0 0}, upload=446444]: Uploading piece... Jul 16 15:26:42.168796830 (558h16m16.939438165s) [noise=true , address=77.90.31.18:4805 , piece_id=S5Q5RUDX7XGQCFFO6VN6HO2KBDZ4ZAONC63OZY6UVGMJ3EOUBXWQ, seg_pos={0 0}, upload=446444]: Uploading piece... Jul 16 15:26:42.169068480 (558h16m16.939709815s) [noise=true , address=104.36.82.10:4078 , piece_id=APH2DW3XCGM5V35P2CWYYYNCIJBHXIUJFHBGCIAO7TXG7T3CGK2Q, seg_pos={0 0}, upload=446444]: Uploading piece... Jul 16 15:26:42.169079261 (558h16m16.939720596s) [noise=true , address=77.90.31.17:4771 , piece_id=NL2IFAFZP7LQAJOQYNKQ6DQZSN4WJFS7UQBOZZBMIFLOM3EA3WJA, seg_pos={0 0}, upload=446444]: Uploading piece... Jul 16 15:26:42.169299281 (558h16m16.939940617s) [noise=true , address=77.90.31.19:4842 , piece_id=4BSORD7ZOFT6Z56UQIXVR2V23G2YIER5UFPGM4KOLZOMJXD4XJ7A, seg_pos={0 0}, upload=446444]: Uploading piece... Jul 16 15:26:42.169578976 (558h16m16.940220322s) [noise=true , address=45.91.116.12:4226 , piece_id=5E2G4BELY3CJ7U62Z4O3A4M2UG7PXHCJU5JS6JSVUZL2XBQWKOOQ, seg_pos={0 0}, upload=446444]: Uploading piece... Jul 16 15:26:42.170038456 (558h16m16.940679791s) [noise=true , address=104.36.82.9:4007 , piece_id=KWE5H32YNYWAXIX4QHL47MHLEBN26NRLGUZU2TH23DP3G3PWWJOQ, seg_pos={0 0}, upload=446444]: Uploading piece... Jul 16 15:26:42.170403254 (558h16m16.941044589s) [noise=true , address=77.90.31.16:4733 , piece_id=TNFR32M4XUYBR6O6JS43JXUMUVDNH7OR5CS2SVKPU6MEVZFDIHNQ, seg_pos={0 0}, upload=446444]: Uploading piece... Jul 16 15:26:43.915440290 (558h16m18.686081625s) [upload=446444]: Next returned nil segment Jul 16 15:26:43.915461170 (558h16m18.686102505s) [upload=446444]: Waiting for error group managing segments... Jul 16 15:26:44.210947937 (558h16m18.981589272s) [noise=true , address=104.36.82.9:4007 , piece_id=KWE5H32YNYWAXIX4QHL47MHLEBN26NRLGUZU2TH23DP3G3PWWJOQ, seg_pos={0 0}, upload=446444]: Done uploading piece. err: <nil> Jul 16 15:26:44.218570381 (558h16m18.989211717s) [noise=true , address=45.91.116.9:4261 , piece_id=7PYHMSQIXZOSBOFYVRTCDVWD23HU2LQKPCLN7XSRJWD3KRK64IQQ, seg_pos={0 0}, upload=446444]: Done uploading piece. err: <nil> Jul 16 15:26:44.232997806 (558h16m19.003639141s) [noise=true , address=45.91.116.12:4226 , piece_id=5E2G4BELY3CJ7U62Z4O3A4M2UG7PXHCJU5JS6JSVUZL2XBQWKOOQ, seg_pos={0 0}, upload=446444]: Done uploading piece. err: <nil> Jul 16 15:26:44.234242647 (558h16m19.004883982s) [noise=true , address=45.91.116.10:4081 , piece_id=B5V5HZKLYKGUNMYXTOURMR66NSXNP3ZOI5ZZMMRCEGXFOXR6NMBQ, seg_pos={0 0}, upload=446444]: Done uploading piece. err: <nil> Jul 16 15:26:44.300054788 (558h16m19.070696123s) [noise=true , address=45.91.116.10:4082 , piece_id=LP3SERHWRQOC2JIESAM5JBH6NY4Q3N5WWTPFN4UHJR2PJ5O3OIRA, seg_pos={0 0}, upload=446444]: Done uploading piece. err: <nil> Jul 16 15:26:44.305389285 (558h16m19.07603062s) [noise=true , address=104.36.82.6:4874 , piece_id=H4BZTYSK5C3M2RKCGJFPAUCBMFMUD2TWRXX6NZ7JEGJMWTF7HD7A, seg_pos={0 0}, upload=446444]: Done uploading piece. err: <nil> Jul 16 15:26:44.317261331 (558h16m19.087902656s) [noise=true , address=104.36.82.13:4185 , piece_id=HYKG4PUJRBHFVQBFPSCJU3QBFYZEBWURM4BFQUZIEHIKZMCJQBVQ, seg_pos={0 0}, upload=446444]: Done uploading piece. err: <nil> Jul 16 15:26:44.358227021 (558h16m19.128868346s) [noise=true , address=104.36.82.16:4295 , piece_id=2SGFTI7BERIIRNOSWMHN4TIFBNIEQTHD4ZM2YAGENOI22LC76W4A, seg_pos={0 0}, upload=446444]: Done uploading piece. err: <nil> Jul 16 15:26:44.359676234 (558h16m19.130317569s) [noise=true , address=45.91.116.7:4011 , piece_id=4LTNQL5LI75YNKEID25D2AHJDKIHQNBOMQ7GMGUCHNP3YIEO5ADQ, seg_pos={0 0}, upload=446444]: Done uploading piece. err: <nil> Jul 16 15:26:44.390317993 (558h16m19.160959328s) [noise=true , address=104.36.82.10:4078 , piece_id=APH2DW3XCGM5V35P2CWYYYNCIJBHXIUJFHBGCIAO7TXG7T3CGK2Q, seg_pos={0 0}, upload=446444]: Done uploading piece. err: <nil> Jul 16 15:27:11.071038188 (558h16m45.841679523s) [noise=true , address=77.90.31.17:4771 , piece_id=NL2IFAFZP7LQAJOQYNKQ6DQZSN4WJFS7UQBOZZBMIFLOM3EA3WJA, seg_pos={0 0}, upload=446444]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:57674->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:57674->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57674->77.90.31.17:4771: use of closed network connection Jul 16 15:27:11.071082664 (558h16m45.841723999s) [noise=true , address=77.90.31.16:4733 , piece_id=TNFR32M4XUYBR6O6JS43JXUMUVDNH7OR5CS2SVKPU6MEVZFDIHNQ, seg_pos={0 0}, upload=446444]: Done uploading piece. err: ecclient: upload failed (node:12TBBdm1hfKVn49KrFVNj3MRbqoanGxJW82ws1iJqMtjRdzJKVf, address:77.90.31.16:4733): protocol: write tcp 188.166.225.170:41338->77.90.31.16:4733: use of closed network connection; write tcp 188.166.225.170:41338->77.90.31.16:4733: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41338->77.90.31.16:4733: use of closed network connection Jul 16 15:27:11.071094787 (558h16m45.841736122s) [noise=true , address=77.90.31.11:4554 , piece_id=CAV37AKARHLV36PGYWCJ6LQML6RNFM6QYMC232PZAE2TNCZJNGWQ, seg_pos={0 0}, upload=446444]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:42614->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:42614->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42614->77.90.31.11:4554: use of closed network connection Jul 16 15:27:11.071114985 (558h16m45.841756321s) [noise=true , address=77.90.31.13:4627 , piece_id=VFJIJMDVJA62SW55JUGRYB67DP3WQ2OFB5ZB6BLHZFE3NCYERISA, seg_pos={0 0}, upload=446444]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:54200->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:54200->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54200->77.90.31.13:4627: use of closed network connection Jul 16 15:27:11.071122840 (558h16m45.841764175s) [noise=true , address=77.90.31.19:4842 , piece_id=4BSORD7ZOFT6Z56UQIXVR2V23G2YIER5UFPGM4KOLZOMJXD4XJ7A, seg_pos={0 0}, upload=446444]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: write tcp 188.166.225.170:49914->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:49914->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49914->77.90.31.19:4842: use of closed network connection Jul 16 15:27:11.071129894 (558h16m45.841771229s) [noise=true , address=77.90.31.18:4805 , piece_id=S5Q5RUDX7XGQCFFO6VN6HO2KBDZ4ZAONC63OZY6UVGMJ3EOUBXWQ, seg_pos={0 0}, upload=446444]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: write tcp 188.166.225.170:37346->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:37346->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37346->77.90.31.18:4805: use of closed network connection Jul 16 15:27:11.071145774 (558h16m45.841787109s) [seg_pos={0 0}, upload=446444]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 16 15:27:11.302813774 (558h16m46.073455109s) [upload=446444]: Done uploading segments... Jul 16 15:27:11.302841668 (558h16m46.073483023s) [upload=446444]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.233677807s core uplink us-select-1
checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 911.082962ms remote: "tcp:5.161.37.179:443" response headers: Content-Length=0 Date=Wed, 16 Jul 2025 15:03:45 GMT Access-Control-Allow-Origin=* target: https://www.storj-ipfs.com/api/v0/add FAIL (2025-07-16 15:26:55 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "" Duration: 911.125573ms edge ipfs us1
checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 924.512236ms remote: "tcp:5.161.37.179:443" response headers: Content-Length=0 Date=Wed, 16 Jul 2025 15:03:45 GMT Access-Control-Allow-Origin=* target: https://www.storj-ipfs.com/api/v0/add FAIL (2025-07-16 15:26:40 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "" Duration: 924.530942ms edge ipfs us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [......................................................X..XX.] duration: 5.790417221s FAIL (2025-07-16 15:26:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.790433672s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.......................................................X..XX] duration: 1m0.238766074s FAIL (2025-07-16 15:26:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.238783728s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 30.231187695s FAIL (2025-07-16 15:26:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ) [upload=446417]: Done uploading segments... Jul 16 15:24:53.740355234 (558h14m28.51099657s) [upload=446417]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Jul 16 15:25:34.710661332 (558h15m9.481302667s) [upload=446429]: Starting upload Jul 16 15:25:34.710714825 (558h15m9.48135616s) [upload=446429]: Done starting upload Jul 16 15:25:34.710765291 (558h15m9.481406626s) [upload=446429]: Uploading segments... Jul 16 15:25:34.710882625 (558h15m9.481523961s) [upload=446429]: Got next segment. Inline: false Jul 16 15:25:34.710908235 (558h15m9.48154957s) [upload=446429]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:25:34.710912202 (558h15m9.481553537s) [upload=446429]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:25:34.944191865 (558h15m9.71483324s) [seg_pos={0 0}, upload=446429]: Begin upload segment... Jul 16 15:25:34.944245888 (558h15m9.714887253s) [seg_pos={0 0}, upload=446429]: Done begin upload segment. Jul 16 15:25:34.944254885 (558h15m9.71489622s) [noise=true , address=104.36.82.5:4838 , piece_id=QQWJRVTZ4E5J2XUZV2NC5FTB2PBTIADW3RNIR3GMUQTTSREB4A5Q, seg_pos={0 0}, upload=446429]: Uploading piece... Jul 16 15:25:34.944275104 (558h15m9.714916469s) [noise=true , address=104.36.82.9:4005 , piece_id=W3WIYQ54JIFK5PD74UH6LDRRKFSZZDPXOY5W3EJUX6BWJB2CWMXA, seg_pos={0 0}, upload=446429]: Uploading piece... Jul 16 15:25:34.945033646 (558h15m9.715675041s) [noise=true , address=45.91.116.6:4974 , piece_id=UC5P3JQ3DQR76X2GDCSU7W4GNY6Z3V6NCALMLYJF67E6ENURRZHA, seg_pos={0 0}, upload=446429]: Uploading piece... Jul 16 15:25:34.945247404 (558h15m9.715888739s) [noise=true , address=104.36.82.11:4115 , piece_id=YZGNIPVMT4QM6KQZBIMB32JHWT5PLSVFDROYLWX4GSHAI2HE55EQ, seg_pos={0 0}, upload=446429]: Uploading piece... Jul 16 15:25:34.945527371 (558h15m9.716168696s) [noise=true , address=104.36.82.13:4185 , piece_id=QSNL5WYKRBEYBQ3U5SJLX6JARAPIHVYRFGFFVTL23EYH6LTU7MMQ, seg_pos={0 0}, upload=446429]: Uploading piece... Jul 16 15:25:34.945681996 (558h15m9.716323331s) [noise=true , address=104.36.82.6:4874 , piece_id=RTXS2UGYXQF3OWNXOS62YGLTMK5TFTAOW53DXOMXUHEH2OTMLO4A, seg_pos={0 0}, upload=446429]: Uploading piece... Jul 16 15:25:34.945724858 (558h15m9.716366183s) [noise=true , address=104.36.82.16:4295 , piece_id=EEZ6BQ4B3PT75FOKDUDKC44UDNVDCYZTOYDE4HRPJLUISHOGMM4Q, seg_pos={0 0}, upload=446429]: Uploading piece... Jul 16 15:25:34.945941663 (558h15m9.716582998s) [noise=true , address=77.90.31.15:4699 , piece_id=WYV77HJKXBROJLPG2ZBKHTBEBZHISVD75D5XIPQUMPIXZMTGG5KA, seg_pos={0 0}, upload=446429]: Uploading piece... Jul 16 15:25:34.946057535 (558h15m9.71669887s) [noise=true , address=104.36.82.8:4969 , piece_id=JMNLB7YWNMADALGRC6WUGZ626UZFRHRTFKGXMSEJH5HLBCCDN3CQ, seg_pos={0 0}, upload=446429]: Uploading piece... Jul 16 15:25:34.946139632 (558h15m9.716780967s) [noise=true , address=45.91.116.13:4155 , piece_id=273TAKMF5P2YURLUDVJL5QKRSX776X6IZYX5C2QJ5O6DBODCWDVA, seg_pos={0 0}, upload=446429]: Uploading piece... Jul 16 15:25:34.946374450 (558h15m9.717015775s) [noise=true , address=77.90.31.12:4589 , piece_id=VXEO6HTVDXMO2KDJYZ7THLV6XZ2BALIPGQAIPFX4U6AT4XUGVMVQ, seg_pos={0 0}, upload=446429]: Uploading piece... Jul 16 15:25:34.946441609 (558h15m9.717082945s) [noise=true , address=104.36.82.12:4150 , piece_id=TTI5EGCDC2M5JU5A4HAKHS4GYQPO5YXICNPARYN2R7SED3MGIAAQ, seg_pos={0 0}, upload=446429]: Uploading piece... Jul 16 15:25:34.946563542 (558h15m9.717204867s) [noise=true , address=45.91.116.14:4190 , piece_id=HKFPL5K5F36UXWO3S5SCZLRONV4LGQ4WD5PMAGYAEP2WQ3242GAA, seg_pos={0 0}, upload=446429]: Uploading piece... Jul 16 15:25:34.946745190 (558h15m9.717386525s) [noise=true , address=77.90.31.17:4770 , piece_id=YBU5XK32VMBZ7JX36MC455EESX752XYLAF5YC456URV4RD342UFQ, seg_pos={0 0}, upload=446429]: Uploading piece... Jul 16 15:25:34.946961164 (558h15m9.717602489s) [noise=true , address=45.91.116.12:4227 , piece_id=QQG2ZO22IPG5R5ZYR6BC4FKRIKEIZFATUMKAG56CKWR2DVN73HWQ, seg_pos={0 0}, upload=446429]: Uploading piece... Jul 16 15:25:34.947168219 (558h15m9.717809554s) [noise=true , address=77.90.31.18:4807 , piece_id=CQTOX53M6LNET57JAF3XVHPL7AAAMKVVHVLGW4SKHPA5PWJSACTQ, seg_pos={0 0}, upload=446429]: Uploading piece... Jul 16 15:25:36.596231347 (558h15m11.366872682s) [upload=446429]: Next returned nil segment Jul 16 15:25:36.596248169 (558h15m11.366889504s) [upload=446429]: Waiting for error group managing segments... Jul 16 15:25:36.993078961 (558h15m11.763720296s) [noise=true , address=104.36.82.8:4969 , piece_id=JMNLB7YWNMADALGRC6WUGZ626UZFRHRTFKGXMSEJH5HLBCCDN3CQ, seg_pos={0 0}, upload=446429]: Done uploading piece. err: <nil> Jul 16 15:25:37.009124151 (558h15m11.779765506s) [noise=true , address=45.91.116.14:4190 , piece_id=HKFPL5K5F36UXWO3S5SCZLRONV4LGQ4WD5PMAGYAEP2WQ3242GAA, seg_pos={0 0}, upload=446429]: Done uploading piece. err: <nil> Jul 16 15:25:37.024103001 (558h15m11.794744326s) [noise=true , address=104.36.82.11:4115 , piece_id=YZGNIPVMT4QM6KQZBIMB32JHWT5PLSVFDROYLWX4GSHAI2HE55EQ, seg_pos={0 0}, upload=446429]: Done uploading piece. err: <nil> Jul 16 15:25:37.036444554 (558h15m11.807085889s) [noise=true , address=45.91.116.6:4974 , piece_id=UC5P3JQ3DQR76X2GDCSU7W4GNY6Z3V6NCALMLYJF67E6ENURRZHA, seg_pos={0 0}, upload=446429]: Done uploading piece. err: <nil> Jul 16 15:25:37.045085207 (558h15m11.815726542s) [noise=true , address=104.36.82.6:4874 , piece_id=RTXS2UGYXQF3OWNXOS62YGLTMK5TFTAOW53DXOMXUHEH2OTMLO4A, seg_pos={0 0}, upload=446429]: Done uploading piece. err: <nil> Jul 16 15:25:37.066480844 (558h15m11.837122169s) [noise=true , address=104.36.82.5:4838 , piece_id=QQWJRVTZ4E5J2XUZV2NC5FTB2PBTIADW3RNIR3GMUQTTSREB4A5Q, seg_pos={0 0}, upload=446429]: Done uploading piece. err: <nil> Jul 16 15:25:37.093394819 (558h15m11.864036154s) [noise=true , address=104.36.82.16:4295 , piece_id=EEZ6BQ4B3PT75FOKDUDKC44UDNVDCYZTOYDE4HRPJLUISHOGMM4Q, seg_pos={0 0}, upload=446429]: Done uploading piece. err: <nil> Jul 16 15:25:37.128677076 (558h15m11.899318401s) [noise=true , address=104.36.82.9:4005 , piece_id=W3WIYQ54JIFK5PD74UH6LDRRKFSZZDPXOY5W3EJUX6BWJB2CWMXA, seg_pos={0 0}, upload=446429]: Done uploading piece. err: <nil> Jul 16 15:25:37.145060464 (558h15m11.915701799s) [noise=true , address=104.36.82.13:4185 , piece_id=QSNL5WYKRBEYBQ3U5SJLX6JARAPIHVYRFGFFVTL23EYH6LTU7MMQ, seg_pos={0 0}, upload=446429]: Done uploading piece. err: <nil> Jul 16 15:25:37.149236544 (558h15m11.919877879s) [noise=true , address=104.36.82.12:4150 , piece_id=TTI5EGCDC2M5JU5A4HAKHS4GYQPO5YXICNPARYN2R7SED3MGIAAQ, seg_pos={0 0}, upload=446429]: Done uploading piece. err: <nil> Jul 16 15:25:37.376702563 (558h15m12.147343898s) [noise=true , address=45.91.116.12:4227 , piece_id=QQG2ZO22IPG5R5ZYR6BC4FKRIKEIZFATUMKAG56CKWR2DVN73HWQ, seg_pos={0 0}, upload=446429]: Done uploading piece. err: <nil> Jul 16 15:25:38.379753720 (558h15m13.150395075s) [noise=true , address=45.91.116.13:4155 , piece_id=273TAKMF5P2YURLUDVJL5QKRSX776X6IZYX5C2QJ5O6DBODCWDVA, seg_pos={0 0}, upload=446429]: Done uploading piece. err: <nil> Jul 16 15:26:04.065959895 (558h15m38.83660129s) [noise=true , address=77.90.31.18:4807 , piece_id=CQTOX53M6LNET57JAF3XVHPL7AAAMKVVHVLGW4SKHPA5PWJSACTQ, seg_pos={0 0}, upload=446429]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:32962->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:32962->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32962->77.90.31.18:4807: use of closed network connection Jul 16 15:26:04.065981597 (558h15m38.836622932s) [noise=true , address=77.90.31.17:4770 , piece_id=YBU5XK32VMBZ7JX36MC455EESX752XYLAF5YC456URV4RD342UFQ, seg_pos={0 0}, upload=446429]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: write tcp 188.166.225.170:38656->77.90.31.17:4770: use of closed network connection; write tcp 188.166.225.170:38656->77.90.31.17:4770: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38656->77.90.31.17:4770: use of closed network connection Jul 16 15:26:04.066063564 (558h15m38.836704909s) [noise=true , address=77.90.31.15:4699 , piece_id=WYV77HJKXBROJLPG2ZBKHTBEBZHISVD75D5XIPQUMPIXZMTGG5KA, seg_pos={0 0}, upload=446429]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 188.166.225.170:42654->77.90.31.15:4699: use of closed network connection; write tcp 188.166.225.170:42654->77.90.31.15:4699: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42654->77.90.31.15:4699: use of closed network connection Jul 16 15:26:04.066071990 (558h15m38.836713405s) [noise=true , address=77.90.31.12:4589 , piece_id=VXEO6HTVDXMO2KDJYZ7THLV6XZ2BALIPGQAIPFX4U6AT4XUGVMVQ, seg_pos={0 0}, upload=446429]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54558->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54558->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54558->77.90.31.12:4589: use of closed network connection Jul 16 15:26:04.066091177 (558h15m38.836732592s) [seg_pos={0 0}, upload=446429]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 16 15:26:04.295440789 (558h15m39.066082124s) [upload=446429]: Done uploading segments... Jul 16 15:26:04.295466659 (558h15m39.066107994s) [upload=446429]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.231209277s core uplink us-select-1
checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 907.085734ms remote: "tcp:5.161.37.179:443" response headers: Content-Length=0 Date=Wed, 16 Jul 2025 15:03:45 GMT Access-Control-Allow-Origin=* target: https://www.storj-ipfs.com/api/v0/add FAIL (2025-07-16 15:25:43 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "" Duration: 907.112996ms edge ipfs us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........................................................X..X] duration: 1m0.234077177s FAIL (2025-07-16 15:25:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== :26.539278126 (558h14m1.309919461s) [noise=true , address=104.36.82.40:4951 , piece_id=KAYJ6F44XF3DEWFWK246CMQHZNVX5YHABG4FBORNFQY23EA5MAVA, seg_pos={0 0}, upload=446419]: Uploading piece... Jul 16 15:24:26.539656159 (558h14m1.310297494s) [noise=true , address=77.90.31.17:4769 , piece_id=L2UPOCH7N5Y6GF64IYBTY6ST72P35LGOQCVRA4KBG7SHYSZ3JV4Q, seg_pos={0 0}, upload=446419]: Uploading piece... Jul 16 15:24:26.539661269 (558h14m1.310302604s) [noise=true , address=45.91.116.8:4078 , piece_id=M27HCQZU7QNO5V7YEUQ6KIR2JWGJVCUXEWGE6SZKEIFYPWBQFV5Q, seg_pos={0 0}, upload=446419]: Uploading piece... Jul 16 15:24:26.540217423 (558h14m1.310858758s) [noise=true , address=104.36.82.7:4927 , piece_id=7QFPWNP7JQ3GOVRU6NFFGBGITMJHQBERU5OTWKCW2QQ2FSRBMVFQ, seg_pos={0 0}, upload=446419]: Uploading piece... Jul 16 15:24:26.540717750 (558h14m1.311359085s) [noise=true , address=45.91.116.5:4316 , piece_id=PHRF3XBYHEKKDVV63QIXHVH5FPV3PASWKL2VEOOKSBB2ZLOILNKQ, seg_pos={0 0}, upload=446419]: Uploading piece... Jul 16 15:24:26.541156970 (558h14m1.311798305s) [noise=true , address=77.90.31.15:4716 , piece_id=6FG6RUG2HHAV2MSHA3XIB454T5CSR5FLC5XAFXSJFRZFCMF5C7BQ, seg_pos={0 0}, upload=446419]: Uploading piece... Jul 16 15:24:26.541420966 (558h14m1.312062301s) [noise=true , address=77.90.31.21:4892 , piece_id=ROGD7SPJZJXB7FLSQNZTQZG36RSAENHPW5QT5FFNZ3BDH3F3VSBQ, seg_pos={0 0}, upload=446419]: Uploading piece... Jul 16 15:24:26.541675012 (558h14m1.312316347s) [noise=true , address=45.91.116.6:4986 , piece_id=RN3DOLUSXTHEIJQN5DCZF6XO4KQFR5S7HIQVIP5FL7SDBOAUQPMA, seg_pos={0 0}, upload=446419]: Uploading piece... Jul 16 15:24:26.541980626 (558h14m1.312621961s) [noise=true , address=77.90.31.18:4816 , piece_id=IIVDIZBIQJT4GFWBRZZOOF72BUAQ232J4WKDQUB72Z6K3N6C55LA, seg_pos={0 0}, upload=446419]: Uploading piece... Jul 16 15:24:26.542234402 (558h14m1.312875737s) [noise=true , address=45.91.95.4:4028 , piece_id=E3MJKI4OINZSW7TESKGET7RVVRO5LTSPUYI3EA4ME2HJE7XS2MVQ, seg_pos={0 0}, upload=446419]: Uploading piece... Jul 16 15:24:26.542507184 (558h14m1.313148519s) [noise=true , address=79.127.219.51:4185 , piece_id=T3GAM4QCJGPC3VDUEMMPINJFA6Z23MMIGLFCL7CBAVHO4H6K7XOA, seg_pos={0 0}, upload=446419]: Uploading piece... Jul 16 15:24:26.542818951 (558h14m1.313460296s) [noise=true , address=77.90.31.10:4541 , piece_id=MGMCUNEBP6HGS24Z4FVZI4MYNV6FN3YGXL5YWNNVXCIWZLDGKXRQ, seg_pos={0 0}, upload=446419]: Uploading piece... Jul 16 15:24:26.544978864 (558h14m1.315620199s) [noise=true , address=79.127.220.101:4041 , piece_id=BIPJ5PJFHNMZERTSDXV4TQAEQ2GREXFGFBE2NYFDZNFF4FJD5MEA, seg_pos={0 0}, upload=446419]: Uploading piece... Jul 16 15:24:26.545536821 (558h14m1.316178156s) [noise=true , address=45.91.116.4:4966 , piece_id=MHV2NG2B736QTSQ7X434KZELS46FPLDONX3DMGLQ4IBUANCDRKWA, seg_pos={0 0}, upload=446419]: Uploading piece... Jul 16 15:24:26.545980690 (558h14m1.316622025s) [noise=true , address=77.90.31.18:4826 , piece_id=WGJH7QZHKSUK3WPPIXE3SPOEQVZAIZNC2FGOP2VV2UXGMJ3CSC3Q, seg_pos={0 0}, upload=446419]: Uploading piece... Jul 16 15:24:26.546460909 (558h14m1.317102245s) [noise=true , address=45.91.116.10:4114 , piece_id=IL7XABB23QJGEHBEFDUZLO5MMDLX2DNV76PWEJUNSPPYVZOHWO6Q, seg_pos={0 0}, upload=446419]: Uploading piece... Jul 16 15:24:26.546738871 (558h14m1.317380206s) [noise=true , address=45.91.116.13:4186 , piece_id=2AD7CIWOWTF7JQHWDPYBWPX6OFCHIGD2GSOUIPEKXMILIYUI2GUA, seg_pos={0 0}, upload=446419]: Uploading piece... Jul 16 15:24:26.546982016 (558h14m1.317623351s) [noise=true , address=45.91.95.3:4026 , piece_id=GU5UEUNNW7L7K5W656SELSGE3USG7QG3SWF6OC67RFJABSHMICWQ, seg_pos={0 0}, upload=446419]: Uploading piece... Jul 16 15:24:27.879178079 (558h14m2.649819414s) [upload=446419]: Next returned nil segment Jul 16 15:24:27.879198518 (558h14m2.649839843s) [upload=446419]: Waiting for error group managing segments... Jul 16 15:24:28.231161112 (558h14m3.001802447s) [noise=true , address=45.91.95.4:4028 , piece_id=E3MJKI4OINZSW7TESKGET7RVVRO5LTSPUYI3EA4ME2HJE7XS2MVQ, seg_pos={0 0}, upload=446419]: Done uploading piece. err: <nil> Jul 16 15:24:28.262179550 (558h14m3.032820885s) [noise=true , address=45.91.95.3:4026 , piece_id=GU5UEUNNW7L7K5W656SELSGE3USG7QG3SWF6OC67RFJABSHMICWQ, seg_pos={0 0}, upload=446419]: Done uploading piece. err: <nil> Jul 16 15:24:28.286342553 (558h14m3.056983888s) [noise=true , address=79.127.220.101:4041 , piece_id=BIPJ5PJFHNMZERTSDXV4TQAEQ2GREXFGFBE2NYFDZNFF4FJD5MEA, seg_pos={0 0}, upload=446419]: Done uploading piece. err: <nil> Jul 16 15:24:28.477336229 (558h14m3.247977564s) [noise=true , address=45.91.116.6:4986 , piece_id=RN3DOLUSXTHEIJQN5DCZF6XO4KQFR5S7HIQVIP5FL7SDBOAUQPMA, seg_pos={0 0}, upload=446419]: Done uploading piece. err: <nil> Jul 16 15:24:28.477634860 (558h14m3.248276185s) [noise=true , address=45.91.116.4:4966 , piece_id=MHV2NG2B736QTSQ7X434KZELS46FPLDONX3DMGLQ4IBUANCDRKWA, seg_pos={0 0}, upload=446419]: Done uploading piece. err: <nil> Jul 16 15:24:28.537269581 (558h14m3.307910907s) [noise=true , address=45.91.116.13:4186 , piece_id=2AD7CIWOWTF7JQHWDPYBWPX6OFCHIGD2GSOUIPEKXMILIYUI2GUA, seg_pos={0 0}, upload=446419]: Done uploading piece. err: <nil> Jul 16 15:24:28.542578459 (558h14m3.313219794s) [noise=true , address=104.36.82.7:4927 , piece_id=7QFPWNP7JQ3GOVRU6NFFGBGITMJHQBERU5OTWKCW2QQ2FSRBMVFQ, seg_pos={0 0}, upload=446419]: Done uploading piece. err: <nil> Jul 16 15:24:28.545411509 (558h14m3.316052844s) [noise=true , address=79.127.219.51:4185 , piece_id=T3GAM4QCJGPC3VDUEMMPINJFA6Z23MMIGLFCL7CBAVHO4H6K7XOA, seg_pos={0 0}, upload=446419]: Done uploading piece. err: <nil> Jul 16 15:24:28.574532317 (558h14m3.345173652s) [noise=true , address=104.36.82.40:4951 , piece_id=KAYJ6F44XF3DEWFWK246CMQHZNVX5YHABG4FBORNFQY23EA5MAVA, seg_pos={0 0}, upload=446419]: Done uploading piece. err: <nil> Jul 16 15:24:28.605076629 (558h14m3.375717954s) [noise=true , address=45.91.116.5:4316 , piece_id=PHRF3XBYHEKKDVV63QIXHVH5FPV3PASWKL2VEOOKSBB2ZLOILNKQ, seg_pos={0 0}, upload=446419]: Done uploading piece. err: <nil> Jul 16 15:24:28.606654198 (558h14m3.377295533s) [noise=true , address=45.91.116.8:4078 , piece_id=M27HCQZU7QNO5V7YEUQ6KIR2JWGJVCUXEWGE6SZKEIFYPWBQFV5Q, seg_pos={0 0}, upload=446419]: Done uploading piece. err: <nil> Jul 16 15:24:28.738458443 (558h14m3.509099778s) [noise=true , address=45.91.116.10:4114 , piece_id=IL7XABB23QJGEHBEFDUZLO5MMDLX2DNV76PWEJUNSPPYVZOHWO6Q, seg_pos={0 0}, upload=446419]: Done uploading piece. err: <nil> Jul 16 15:25:20.194900146 (558h14m54.965541491s) [noise=true , address=77.90.31.18:4826 , piece_id=WGJH7QZHKSUK3WPPIXE3SPOEQVZAIZNC2FGOP2VV2UXGMJ3CSC3Q, seg_pos={0 0}, upload=446419]: Done uploading piece. err: <nil> Jul 16 15:25:25.434841382 (558h15m0.205482727s) [noise=true , address=77.90.31.18:4816 , piece_id=IIVDIZBIQJT4GFWBRZZOOF72BUAQ232J4WKDQUB72Z6K3N6C55LA, seg_pos={0 0}, upload=446419]: Done uploading piece. err: ecclient: upload failed (node:12Vno3GkFXR7KdTUEAhvesqif7KBF6DZoeMTv8n9MpRbYZNktaT, address:77.90.31.18:4816): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:25:25.435066843 (558h15m0.205708178s) [noise=true , address=77.90.31.17:4769 , piece_id=L2UPOCH7N5Y6GF64IYBTY6ST72P35LGOQCVRA4KBG7SHYSZ3JV4Q, seg_pos={0 0}, upload=446419]: Done uploading piece. err: ecclient: upload failed (node:12DTwUoE6vkVP1pChp2oyoKm47tYCRkfYQXbEKSSH22RxdVxmd, address:77.90.31.17:4769): protocol: write tcp 188.166.225.170:49198->77.90.31.17:4769: use of closed network connection; write tcp 188.166.225.170:49198->77.90.31.17:4769: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49198->77.90.31.17:4769: use of closed network connection Jul 16 15:25:25.435124543 (558h15m0.205765878s) [noise=true , address=77.90.31.10:4541 , piece_id=MGMCUNEBP6HGS24Z4FVZI4MYNV6FN3YGXL5YWNNVXCIWZLDGKXRQ, seg_pos={0 0}, upload=446419]: Done uploading piece. err: ecclient: upload failed (node:1277g6Fg28bEwNXpQEsMvLGSfggDDNn5wPHrkv4inaVD8j71rwW, address:77.90.31.10:4541): protocol: write tcp 188.166.225.170:57926->77.90.31.10:4541: use of closed network connection; write tcp 188.166.225.170:57926->77.90.31.10:4541: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57926->77.90.31.10:4541: use of closed network connection Jul 16 15:25:25.435188495 (558h15m0.20582983s) [noise=true , address=77.90.31.21:4892 , piece_id=ROGD7SPJZJXB7FLSQNZTQZG36RSAENHPW5QT5FFNZ3BDH3F3VSBQ, seg_pos={0 0}, upload=446419]: Done uploading piece. err: ecclient: upload failed (node:1q5ZaXkQ2kzSdKoqNhFzyNuYbWZ2uQern3umD3HtDYgTCSkGAa, address:77.90.31.21:4892): protocol: write tcp 188.166.225.170:51390->77.90.31.21:4892: use of closed network connection; write tcp 188.166.225.170:51390->77.90.31.21:4892: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51390->77.90.31.21:4892: use of closed network connection Jul 16 15:25:25.435193736 (558h15m0.205835071s) [noise=true , address=77.90.31.15:4716 , piece_id=6FG6RUG2HHAV2MSHA3XIB454T5CSR5FLC5XAFXSJFRZFCMF5C7BQ, seg_pos={0 0}, upload=446419]: Done uploading piece. err: ecclient: upload failed (node:12tuEGhEzrpo1eV7v2QatHXHMy7b1S5gcVSymWYWJSoCFUMGnaL, address:77.90.31.15:4716): protocol: write tcp 188.166.225.170:35296->77.90.31.15:4716: use of closed network connection; write tcp 188.166.225.170:35296->77.90.31.15:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35296->77.90.31.15:4716: use of closed network connection Jul 16 15:25:25.435223212 (558h15m0.205864548s) [seg_pos={0 0}, upload=446419]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 16 15:25:25.667064803 (558h15m0.437706138s) [upload=446419]: Done uploading segments... Jul 16 15:25:25.667089400 (558h15m0.437730735s) [upload=446419]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.234092406s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.251119375s FAIL (2025-07-16 15:24:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== 16 15:24:24.155908521 (558h13m58.926549846s) [upload=446417]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:24:24.395728508 (558h13m59.166369853s) [seg_pos={0 0}, upload=446417]: Begin upload segment... Jul 16 15:24:24.395774175 (558h13m59.16641551s) [seg_pos={0 0}, upload=446417]: Done begin upload segment. Jul 16 15:24:24.395798211 (558h13m59.166439547s) [noise=true , address=45.91.116.13:4155 , piece_id=G7GESLHYJ3LB5Q6ZQBO76S7UXT2FYUXNB2XKJOYNBM4LXBBXVXRQ, seg_pos={0 0}, upload=446417]: Uploading piece... Jul 16 15:24:24.395879237 (558h13m59.166520572s) [noise=true , address=45.91.116.7:4010 , piece_id=27IRNBVEJJCIU4URTR5J62JKKYDTH7CX3XGIMRZE6H4SF7CRFROQ, seg_pos={0 0}, upload=446417]: Uploading piece... Jul 16 15:24:24.396132481 (558h13m59.166773816s) [noise=true , address=45.91.116.5:4298 , piece_id=KDAN76QPK77Q2BY6Q2MDNKFWDP22QA2EVFSZCLX2LRSA7BVTAGVA, seg_pos={0 0}, upload=446417]: Uploading piece... Jul 16 15:24:24.396362380 (558h13m59.167003715s) [noise=true , address=77.90.31.12:4592 , piece_id=64F42YSCRAHWK5PEULWN4VFRGRY2ZEW2MEO5DGAL6FDOP5D57SLQ, seg_pos={0 0}, upload=446417]: Uploading piece... Jul 16 15:24:24.396577502 (558h13m59.167218837s) [noise=true , address=77.90.31.17:4771 , piece_id=WKDM6BH4K6LY3PFV6IHQUHVHT64FX67OBA567RF44DMJ64RV4JLA, seg_pos={0 0}, upload=446417]: Uploading piece... Jul 16 15:24:24.396755904 (558h13m59.167397239s) [noise=true , address=45.91.116.10:4083 , piece_id=V4PDXN2M5VM423FO452OGPPROF7HIY54ABTZL5PCMIYCQHTP6BKQ, seg_pos={0 0}, upload=446417]: Uploading piece... Jul 16 15:24:24.396826158 (558h13m59.167467493s) [noise=true , address=45.91.116.9:4261 , piece_id=7QLKKFTFDR5QC5GYM2OFLECJAHY2NGNMXISBES762JCKFBOP3VAQ, seg_pos={0 0}, upload=446417]: Uploading piece... Jul 16 15:24:24.397131021 (558h13m59.167772346s) [noise=true , address=45.91.116.11:4117 , piece_id=CWX7DSN4LZYI43Q3CR4AZ3MXOIX2E7CZOKMLV6WTQSYN7DEBIGKQ, seg_pos={0 0}, upload=446417]: Uploading piece... Jul 16 15:24:24.397258205 (558h13m59.16789955s) [noise=true , address=77.90.31.21:4878 , piece_id=GRPOBRTYRJE5TJXNY3WZOIP2BBAPXM7IRQ6QU2FTYQMSDWTJ4ITA, seg_pos={0 0}, upload=446417]: Uploading piece... Jul 16 15:24:24.397378334 (558h13m59.168019669s) [noise=true , address=77.90.31.14:4661 , piece_id=JNCGQOUXQHUI3U4DTOIX4ICXLG35UKXIHBU55ZMBITNQYYWYVFFQ, seg_pos={0 0}, upload=446417]: Uploading piece... Jul 16 15:24:24.397628643 (558h13m59.168269988s) [noise=true , address=45.91.116.6:4973 , piece_id=5U2T2X4WQ4E46LO6Q7MXKQ2TOMZYOCMW5LMIVP4UHZYCED4QT3TQ, seg_pos={0 0}, upload=446417]: Uploading piece... Jul 16 15:24:24.397756107 (558h13m59.168397442s) [noise=true , address=77.90.31.19:4841 , piece_id=CBAAF4JU3QFAMYSAOOL4G2AG4Q6U6UFVBSZBONUUZ3RNAIVCF42A, seg_pos={0 0}, upload=446417]: Uploading piece... Jul 16 15:24:24.397911765 (558h13m59.168553101s) [noise=true , address=45.91.116.6:4974 , piece_id=HAQS3675GHZJCNXWLKOTPGDXJJGLO754RQQ42LHSDIY5V5O6P42Q, seg_pos={0 0}, upload=446417]: Uploading piece... Jul 16 15:24:24.398188184 (558h13m59.168829519s) [noise=true , address=104.36.82.12:4150 , piece_id=6PFYD2GLLG7NQOCZKDE2PUQGCS6UHF44APFVL7ESNKZXZYSO3RKA, seg_pos={0 0}, upload=446417]: Uploading piece... Jul 16 15:24:24.398619819 (558h13m59.169261154s) [noise=true , address=77.90.31.13:4627 , piece_id=35SVFHU7JIWY6QKRAVO6TK7HQHHV2HYZ2KMSHBFY6XMAZD4JWFSA, seg_pos={0 0}, upload=446417]: Uploading piece... Jul 16 15:24:24.398997773 (558h13m59.169639118s) [noise=true , address=45.91.116.13:4154 , piece_id=QQFQHOZNQYJ7DCMRUL2NQZ53QXQZSX7MABBBOSQZ47UUMHZWSHNQ, seg_pos={0 0}, upload=446417]: Uploading piece... Jul 16 15:24:26.155211727 (558h14m0.925853052s) [upload=446417]: Next returned nil segment Jul 16 15:24:26.155263846 (558h14m0.925905182s) [upload=446417]: Waiting for error group managing segments... Jul 16 15:24:26.436332666 (558h14m1.206974001s) [noise=true , address=45.91.116.13:4154 , piece_id=QQFQHOZNQYJ7DCMRUL2NQZ53QXQZSX7MABBBOSQZ47UUMHZWSHNQ, seg_pos={0 0}, upload=446417]: Done uploading piece. err: <nil> Jul 16 15:24:26.447348674 (558h14m1.217990009s) [noise=true , address=45.91.116.6:4974 , piece_id=HAQS3675GHZJCNXWLKOTPGDXJJGLO754RQQ42LHSDIY5V5O6P42Q, seg_pos={0 0}, upload=446417]: Done uploading piece. err: <nil> Jul 16 15:24:26.450362831 (558h14m1.221004166s) [noise=true , address=104.36.82.12:4150 , piece_id=6PFYD2GLLG7NQOCZKDE2PUQGCS6UHF44APFVL7ESNKZXZYSO3RKA, seg_pos={0 0}, upload=446417]: Done uploading piece. err: <nil> Jul 16 15:24:26.455871801 (558h14m1.226513136s) [noise=true , address=45.91.116.7:4010 , piece_id=27IRNBVEJJCIU4URTR5J62JKKYDTH7CX3XGIMRZE6H4SF7CRFROQ, seg_pos={0 0}, upload=446417]: Done uploading piece. err: <nil> Jul 16 15:24:26.468303858 (558h14m1.238945193s) [noise=true , address=45.91.116.9:4261 , piece_id=7QLKKFTFDR5QC5GYM2OFLECJAHY2NGNMXISBES762JCKFBOP3VAQ, seg_pos={0 0}, upload=446417]: Done uploading piece. err: <nil> Jul 16 15:24:26.478372203 (558h14m1.249013538s) [noise=true , address=45.91.116.13:4155 , piece_id=G7GESLHYJ3LB5Q6ZQBO76S7UXT2FYUXNB2XKJOYNBM4LXBBXVXRQ, seg_pos={0 0}, upload=446417]: Done uploading piece. err: <nil> Jul 16 15:24:26.534231870 (558h14m1.304873205s) [noise=true , address=45.91.116.10:4083 , piece_id=V4PDXN2M5VM423FO452OGPPROF7HIY54ABTZL5PCMIYCQHTP6BKQ, seg_pos={0 0}, upload=446417]: Done uploading piece. err: <nil> Jul 16 15:24:26.535854786 (558h14m1.306496121s) [noise=true , address=45.91.116.5:4298 , piece_id=KDAN76QPK77Q2BY6Q2MDNKFWDP22QA2EVFSZCLX2LRSA7BVTAGVA, seg_pos={0 0}, upload=446417]: Done uploading piece. err: <nil> Jul 16 15:24:26.558362962 (558h14m1.329004297s) [noise=true , address=45.91.116.11:4117 , piece_id=CWX7DSN4LZYI43Q3CR4AZ3MXOIX2E7CZOKMLV6WTQSYN7DEBIGKQ, seg_pos={0 0}, upload=446417]: Done uploading piece. err: <nil> Jul 16 15:24:26.623010344 (558h14m1.393651679s) [noise=true , address=45.91.116.6:4973 , piece_id=5U2T2X4WQ4E46LO6Q7MXKQ2TOMZYOCMW5LMIVP4UHZYCED4QT3TQ, seg_pos={0 0}, upload=446417]: Done uploading piece. err: <nil> Jul 16 15:24:53.491231426 (558h14m28.261872761s) [noise=true , address=77.90.31.14:4661 , piece_id=JNCGQOUXQHUI3U4DTOIX4ICXLG35UKXIHBU55ZMBITNQYYWYVFFQ, seg_pos={0 0}, upload=446417]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:33242->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:33242->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33242->77.90.31.14:4661: use of closed network connection Jul 16 15:24:53.491306289 (558h14m28.261947624s) [noise=true , address=77.90.31.19:4841 , piece_id=CBAAF4JU3QFAMYSAOOL4G2AG4Q6U6UFVBSZBONUUZ3RNAIVCF42A, seg_pos={0 0}, upload=446417]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: write tcp 188.166.225.170:57734->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:57734->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57734->77.90.31.19:4841: use of closed network connection Jul 16 15:24:53.491365092 (558h14m28.262006427s) [noise=true , address=77.90.31.12:4592 , piece_id=64F42YSCRAHWK5PEULWN4VFRGRY2ZEW2MEO5DGAL6FDOP5D57SLQ, seg_pos={0 0}, upload=446417]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: write tcp 188.166.225.170:53672->77.90.31.12:4592: use of closed network connection; write tcp 188.166.225.170:53672->77.90.31.12:4592: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53672->77.90.31.12:4592: use of closed network connection Jul 16 15:24:53.491414005 (558h14m28.26205534s) [noise=true , address=77.90.31.21:4878 , piece_id=GRPOBRTYRJE5TJXNY3WZOIP2BBAPXM7IRQ6QU2FTYQMSDWTJ4ITA, seg_pos={0 0}, upload=446417]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: read tcp 188.166.225.170:47604->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 16 15:24:53.491423995 (558h14m28.26206533s) [noise=true , address=77.90.31.13:4627 , piece_id=35SVFHU7JIWY6QKRAVO6TK7HQHHV2HYZ2KMSHBFY6XMAZD4JWFSA, seg_pos={0 0}, upload=446417]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:39818->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:39818->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39818->77.90.31.13:4627: use of closed network connection Jul 16 15:24:53.491433492 (558h14m28.262074838s) [noise=true , address=77.90.31.17:4771 , piece_id=WKDM6BH4K6LY3PFV6IHQUHVHT64FX67OBA567RF44DMJ64RV4JLA, seg_pos={0 0}, upload=446417]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 188.166.225.170:38956->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:38956->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38956->77.90.31.17:4771: use of closed network connection Jul 16 15:24:53.491460825 (558h14m28.26210216s) [seg_pos={0 0}, upload=446417]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 16 15:24:53.740323494 (558h14m28.510964869s) [upload=446417]: Done uploading segments... Jul 16 15:24:53.740355234 (558h14m28.51099657s) [upload=446417]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration:* 30.251141406s core uplink us-select-1
checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 890.509708ms remote: "tcp:5.161.37.179:443" response headers: Content-Length=0 Date=Wed, 16 Jul 2025 15:03:45 GMT Access-Control-Allow-Origin=* target: https://www.storj-ipfs.com/api/v0/add FAIL (2025-07-16 15:24:41 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "" Duration: 890.548153ms edge ipfs us1
checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 924.889155ms remote: "tcp:5.161.37.179:443" response headers: Content-Length=0 Date=Wed, 16 Jul 2025 15:03:45 GMT Access-Control-Allow-Origin=* target: https://www.storj-ipfs.com/api/v0/add FAIL (2025-07-16 15:24:16 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "" Duration: 924.936866ms edge ipfs us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 11.199932496s FAIL (2025-07-16 15:23:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 11.015974179s) [upload=446377]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:21:36.245338395 (558h11m11.01597973s) [upload=446377]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:21:36.690946493 (558h11m11.461587828s) [seg_pos={0 0}, upload=446377]: Begin upload segment... Jul 16 15:21:36.690972012 (558h11m11.461613337s) [seg_pos={0 0}, upload=446377]: Done begin upload segment. Jul 16 15:21:36.690986640 (558h11m11.461627966s) [noise=true , address=104.36.82.18:4386 , piece_id=6ZHILKLTD74MZA2M6ZM5R2MDW2BAD3GN7EZOXNI6LXLY7ZVSY2SA, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.691041756 (558h11m11.461683091s) [noise=true , address=104.36.82.12:4154 , piece_id=LN56DTNFBMOGDH3NEEPLNNU65ZZ4DKF3ZNCRHIRODPHGFYGN5PMA, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.691307764 (558h11m11.461949089s) [noise=true , address=45.91.116.13:4159 , piece_id=5B34MTGYOC3BYPDKVMNSQNTYSWZWB56FEPES63E6RJRQ7HIL6GCA, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.691424208 (558h11m11.462065533s) [noise=true , address=45.91.95.3:4036 , piece_id=6WGBB66MJWJ2GBWKQROMZBDLLRHEPBOA5HDRXZIOPIU226ENBJEA, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.691568764 (558h11m11.462210099s) [noise=true , address=45.91.95.5:4013 , piece_id=SJ7TKNQ2G642VVORSNSPL7AV7BUV2JNUODS4QLMUFERBGEQE5TEQ, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.691718871 (558h11m11.462360196s) [noise=true , address=156.146.43.225:4457 , piece_id=IMMKGRXALKFMWNPVB7U3XKQOQFR5USOOJZ4DHQV3GSXAVVGP64FQ, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.691837559 (558h11m11.462478894s) [noise=true , address=77.90.31.19:4851 , piece_id=4OHND2X7VSZ7W25MW63MTNY6ID2I4YHULDZQIQ45ZG5YPBRZO6HA, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.692008656 (558h11m11.462649991s) [noise=true , address=77.90.31.11:4556 , piece_id=IWX3HP7PMMEZJH2BFZM6UQMBS75CCIOZZJKXZS3MYJCY54C4MMHA, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.692094189 (558h11m11.462735524s) [noise=true , address=79.127.219.48:4084 , piece_id=OUD2PZT6H5I5SDTBWAMW6UYNQI4GJQ3WFWO5UNGDB4G7ILVQQADA, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.692290194 (558h11m11.462931529s) [noise=true , address=77.90.31.10:4538 , piece_id=O5776XFIWA75ESMGPMJXUV37RLOG2FPW3CFYR6ZCEQKOM6XQ57LA, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.692332916 (558h11m11.462974241s) [noise=true , address=45.91.116.8:4078 , piece_id=4Q7U5S4YMPXZ3APP5JYF2GXFDC3ZJUQ6HJULDNKUHJ7OUB364SPA, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.692537698 (558h11m11.463179033s) [noise=true , address=45.91.116.9:4293 , piece_id=BHXNYDU3FMQE5ABEPKHNSHB6MXW6KEKTPTGEUPR7JRE2EERQH6OQ, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.692542156 (558h11m11.463183491s) [noise=true , address=77.90.31.15:4727 , piece_id=RHF2PKY3VG7G7RE4ML3YYCZSUGMLDWWYNODHZOK5AHGTEM2QLPOA, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.692741288 (558h11m11.463382613s) [noise=true , address=104.36.82.51:4072 , piece_id=C3OHCFSFJ3B7WW4ZCIGNTOL3XVBHEVHFYW6OIJTZJY44MDTYD5AA, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.692785722 (558h11m11.463427047s) [noise=true , address=77.90.31.21:4894 , piece_id=U5CIB5APFQDRGRR2QPDGSAXXWUOYAVT4W3MK2ON4LIQSSHDDXO2A, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.692982458 (558h11m11.463623794s) [noise=true , address=45.91.116.11:4142 , piece_id=QQGR5PIFCHPRWBUVMVTUKAHAISBLSKZNR6OCIR4NM7RXUJO7UX7A, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.693070959 (558h11m11.463712284s) [noise=true , address=104.36.82.8:4983 , piece_id=4LT7B7JH7KLR43PEOAS4FBCM7GARNRRHGBFXVNOQGDYGQG3ZZUZA, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.694049761 (558h11m11.464691086s) [noise=true , address=45.91.95.2:4011 , piece_id=3IICDUFHRFGRSLUUIB6CZJAU67S2X6FRU63U2MBDUKEFZP4C4HGQ, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:38.008052223 (558h11m12.778693558s) [upload=446377]: Next returned nil segment Jul 16 15:21:38.008070457 (558h11m12.778711792s) [upload=446377]: Waiting for error group managing segments... Jul 16 15:21:38.370664362 (558h11m13.141305697s) [noise=true , address=45.91.95.2:4011 , piece_id=3IICDUFHRFGRSLUUIB6CZJAU67S2X6FRU63U2MBDUKEFZP4C4HGQ, seg_pos={0 0}, upload=446377]: Done uploading piece. err: <nil> Jul 16 15:21:38.394746470 (558h11m13.165387805s) [noise=true , address=45.91.95.5:4013 , piece_id=SJ7TKNQ2G642VVORSNSPL7AV7BUV2JNUODS4QLMUFERBGEQE5TEQ, seg_pos={0 0}, upload=446377]: Done uploading piece. err: <nil> Jul 16 15:21:38.560852907 (558h11m13.331494242s) [noise=true , address=45.91.95.3:4036 , piece_id=6WGBB66MJWJ2GBWKQROMZBDLLRHEPBOA5HDRXZIOPIU226ENBJEA, seg_pos={0 0}, upload=446377]: Done uploading piece. err: <nil> Jul 16 15:21:38.587805277 (558h11m13.358446612s) [noise=true , address=156.146.43.225:4457 , piece_id=IMMKGRXALKFMWNPVB7U3XKQOQFR5USOOJZ4DHQV3GSXAVVGP64FQ, seg_pos={0 0}, upload=446377]: Done uploading piece. err: <nil> Jul 16 15:21:38.597713243 (558h11m13.368354578s) [noise=true , address=45.91.116.8:4078 , piece_id=4Q7U5S4YMPXZ3APP5JYF2GXFDC3ZJUQ6HJULDNKUHJ7OUB364SPA, seg_pos={0 0}, upload=446377]: Done uploading piece. err: <nil> Jul 16 15:21:38.599006327 (558h11m13.369647662s) [noise=true , address=45.91.116.13:4159 , piece_id=5B34MTGYOC3BYPDKVMNSQNTYSWZWB56FEPES63E6RJRQ7HIL6GCA, seg_pos={0 0}, upload=446377]: Done uploading piece. err: <nil> Jul 16 15:21:38.682044158 (558h11m13.452685493s) [noise=true , address=79.127.219.48:4084 , piece_id=OUD2PZT6H5I5SDTBWAMW6UYNQI4GJQ3WFWO5UNGDB4G7ILVQQADA, seg_pos={0 0}, upload=446377]: Done uploading piece. err: <nil> Jul 16 15:21:38.710856095 (558h11m13.48149743s) [noise=true , address=104.36.82.8:4983 , piece_id=4LT7B7JH7KLR43PEOAS4FBCM7GARNRRHGBFXVNOQGDYGQG3ZZUZA, seg_pos={0 0}, upload=446377]: Done uploading piece. err: <nil> Jul 16 15:21:38.724282394 (558h11m13.494923729s) [noise=true , address=104.36.82.12:4154 , piece_id=LN56DTNFBMOGDH3NEEPLNNU65ZZ4DKF3ZNCRHIRODPHGFYGN5PMA, seg_pos={0 0}, upload=446377]: Done uploading piece. err: <nil> Jul 16 15:21:38.736872973 (558h11m13.507514359s) [noise=true , address=104.36.82.18:4386 , piece_id=6ZHILKLTD74MZA2M6ZM5R2MDW2BAD3GN7EZOXNI6LXLY7ZVSY2SA, seg_pos={0 0}, upload=446377]: Done uploading piece. err: <nil> Jul 16 15:21:38.747087357 (558h11m13.517728692s) [noise=true , address=45.91.116.11:4142 , piece_id=QQGR5PIFCHPRWBUVMVTUKAHAISBLSKZNR6OCIR4NM7RXUJO7UX7A, seg_pos={0 0}, upload=446377]: Done uploading piece. err: <nil> Jul 16 15:21:38.780196437 (558h11m13.550837772s) [noise=true , address=104.36.82.51:4072 , piece_id=C3OHCFSFJ3B7WW4ZCIGNTOL3XVBHEVHFYW6OIJTZJY44MDTYD5AA, seg_pos={0 0}, upload=446377]: Done uploading piece. err: <nil> Jul 16 15:21:38.807244329 (558h11m13.577885664s) [noise=true , address=45.91.116.9:4293 , piece_id=BHXNYDU3FMQE5ABEPKHNSHB6MXW6KEKTPTGEUPR7JRE2EERQH6OQ, seg_pos={0 0}, upload=446377]: Done uploading piece. err: <nil> Jul 16 15:22:35.602262422 (558h12m10.372903757s) [noise=true , address=77.90.31.10:4538 , piece_id=O5776XFIWA75ESMGPMJXUV37RLOG2FPW3CFYR6ZCEQKOM6XQ57LA, seg_pos={0 0}, upload=446377]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:22:35.602318459 (558h12m10.372959794s) [noise=true , address=77.90.31.11:4556 , piece_id=IWX3HP7PMMEZJH2BFZM6UQMBS75CCIOZZJKXZS3MYJCY54C4MMHA, seg_pos={0 0}, upload=446377]: Done uploading piece. err: ecclient: upload failed (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, address:77.90.31.11:4556): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:22:35.602534603 (558h12m10.373175938s) [noise=true , address=77.90.31.19:4851 , piece_id=4OHND2X7VSZ7W25MW63MTNY6ID2I4YHULDZQIQ45ZG5YPBRZO6HA, seg_pos={0 0}, upload=446377]: Done uploading piece. err: ecclient: upload failed (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk, address:77.90.31.19:4851): protocol: write tcp 188.166.225.170:39300->77.90.31.19:4851: use of closed network connection; write tcp 188.166.225.170:39300->77.90.31.19:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39300->77.90.31.19:4851: use of closed network connection Jul 16 15:22:35.602591101 (558h12m10.373232436s) [noise=true , address=77.90.31.15:4727 , piece_id=RHF2PKY3VG7G7RE4ML3YYCZSUGMLDWWYNODHZOK5AHGTEM2QLPOA, seg_pos={0 0}, upload=446377]: Done uploading piece. err: ecclient: upload failed (node:1xoAbv6zJuG9LjyCdsCrTWQaCDyLrWyv8RCrMTiqNqazoEwD1M, address:77.90.31.15:4727): protocol: write tcp 188.166.225.170:39106->77.90.31.15:4727: use of closed network connection; write tcp 188.166.225.170:39106->77.90.31.15:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39106->77.90.31.15:4727: use of closed network connection Jul 16 15:22:35.602678709 (558h12m10.373320034s) [noise=true , address=77.90.31.21:4894 , piece_id=U5CIB5APFQDRGRR2QPDGSAXXWUOYAVT4W3MK2ON4LIQSSHDDXO2A, seg_pos={0 0}, upload=446377]: Done uploading piece. err: ecclient: upload failed (node:1JJ81nDtVEinZaEU7NM7AAhCWiFN7V6a8o5Cp9ZzWjvjDCKmMG, address:77.90.31.21:4894): protocol: context deadline exceeded; context deadline exceeded Jul 16 15:22:35.602693527 (558h12m10.373334862s) [seg_pos={0 0}, upload=446377]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 16 15:22:35.835976116 (558h12m10.606617451s) [upload=446377]: Done uploading segments... Jul 16 15:22:35.836002216 (558h12m10.606643551s) [upload=446377]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 11.199959869s core uplink us-select-1 us1-placement-20
checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 942.022297ms remote: "tcp:5.161.37.179:443" response headers: Content-Length=0 Date=Wed, 16 Jul 2025 15:03:45 GMT Access-Control-Allow-Origin=* target: https://www.storj-ipfs.com/api/v0/add FAIL (2025-07-16 15:23:38 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "" Duration: 942.066412ms edge ipfs us1
checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 882.7714ms remote: "tcp:5.161.37.179:443" response headers: Content-Length=0 Date=Wed, 16 Jul 2025 15:03:45 GMT Access-Control-Allow-Origin=* target: https://www.storj-ipfs.com/api/v0/add FAIL (2025-07-16 15:23:19 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "" Duration: 882.793473ms edge ipfs us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.869745063s FAIL (2025-07-16 15:22:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 11.015974179s) [upload=446377]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:21:36.245338395 (558h11m11.01597973s) [upload=446377]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:21:36.690946493 (558h11m11.461587828s) [seg_pos={0 0}, upload=446377]: Begin upload segment... Jul 16 15:21:36.690972012 (558h11m11.461613337s) [seg_pos={0 0}, upload=446377]: Done begin upload segment. Jul 16 15:21:36.690986640 (558h11m11.461627966s) [noise=true , address=104.36.82.18:4386 , piece_id=6ZHILKLTD74MZA2M6ZM5R2MDW2BAD3GN7EZOXNI6LXLY7ZVSY2SA, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.691041756 (558h11m11.461683091s) [noise=true , address=104.36.82.12:4154 , piece_id=LN56DTNFBMOGDH3NEEPLNNU65ZZ4DKF3ZNCRHIRODPHGFYGN5PMA, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.691307764 (558h11m11.461949089s) [noise=true , address=45.91.116.13:4159 , piece_id=5B34MTGYOC3BYPDKVMNSQNTYSWZWB56FEPES63E6RJRQ7HIL6GCA, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.691424208 (558h11m11.462065533s) [noise=true , address=45.91.95.3:4036 , piece_id=6WGBB66MJWJ2GBWKQROMZBDLLRHEPBOA5HDRXZIOPIU226ENBJEA, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.691568764 (558h11m11.462210099s) [noise=true , address=45.91.95.5:4013 , piece_id=SJ7TKNQ2G642VVORSNSPL7AV7BUV2JNUODS4QLMUFERBGEQE5TEQ, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.691718871 (558h11m11.462360196s) [noise=true , address=156.146.43.225:4457 , piece_id=IMMKGRXALKFMWNPVB7U3XKQOQFR5USOOJZ4DHQV3GSXAVVGP64FQ, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.691837559 (558h11m11.462478894s) [noise=true , address=77.90.31.19:4851 , piece_id=4OHND2X7VSZ7W25MW63MTNY6ID2I4YHULDZQIQ45ZG5YPBRZO6HA, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.692008656 (558h11m11.462649991s) [noise=true , address=77.90.31.11:4556 , piece_id=IWX3HP7PMMEZJH2BFZM6UQMBS75CCIOZZJKXZS3MYJCY54C4MMHA, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.692094189 (558h11m11.462735524s) [noise=true , address=79.127.219.48:4084 , piece_id=OUD2PZT6H5I5SDTBWAMW6UYNQI4GJQ3WFWO5UNGDB4G7ILVQQADA, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.692290194 (558h11m11.462931529s) [noise=true , address=77.90.31.10:4538 , piece_id=O5776XFIWA75ESMGPMJXUV37RLOG2FPW3CFYR6ZCEQKOM6XQ57LA, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.692332916 (558h11m11.462974241s) [noise=true , address=45.91.116.8:4078 , piece_id=4Q7U5S4YMPXZ3APP5JYF2GXFDC3ZJUQ6HJULDNKUHJ7OUB364SPA, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.692537698 (558h11m11.463179033s) [noise=true , address=45.91.116.9:4293 , piece_id=BHXNYDU3FMQE5ABEPKHNSHB6MXW6KEKTPTGEUPR7JRE2EERQH6OQ, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.692542156 (558h11m11.463183491s) [noise=true , address=77.90.31.15:4727 , piece_id=RHF2PKY3VG7G7RE4ML3YYCZSUGMLDWWYNODHZOK5AHGTEM2QLPOA, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.692741288 (558h11m11.463382613s) [noise=true , address=104.36.82.51:4072 , piece_id=C3OHCFSFJ3B7WW4ZCIGNTOL3XVBHEVHFYW6OIJTZJY44MDTYD5AA, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.692785722 (558h11m11.463427047s) [noise=true , address=77.90.31.21:4894 , piece_id=U5CIB5APFQDRGRR2QPDGSAXXWUOYAVT4W3MK2ON4LIQSSHDDXO2A, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.692982458 (558h11m11.463623794s) [noise=true , address=45.91.116.11:4142 , piece_id=QQGR5PIFCHPRWBUVMVTUKAHAISBLSKZNR6OCIR4NM7RXUJO7UX7A, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.693070959 (558h11m11.463712284s) [noise=true , address=104.36.82.8:4983 , piece_id=4LT7B7JH7KLR43PEOAS4FBCM7GARNRRHGBFXVNOQGDYGQG3ZZUZA, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.694049761 (558h11m11.464691086s) [noise=true , address=45.91.95.2:4011 , piece_id=3IICDUFHRFGRSLUUIB6CZJAU67S2X6FRU63U2MBDUKEFZP4C4HGQ, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:38.008052223 (558h11m12.778693558s) [upload=446377]: Next returned nil segment Jul 16 15:21:38.008070457 (558h11m12.778711792s) [upload=446377]: Waiting for error group managing segments... Jul 16 15:21:38.370664362 (558h11m13.141305697s) [noise=true , address=45.91.95.2:4011 , piece_id=3IICDUFHRFGRSLUUIB6CZJAU67S2X6FRU63U2MBDUKEFZP4C4HGQ, seg_pos={0 0}, upload=446377]: Done uploading piece. err: <nil> Jul 16 15:21:38.394746470 (558h11m13.165387805s) [noise=true , address=45.91.95.5:4013 , piece_id=SJ7TKNQ2G642VVORSNSPL7AV7BUV2JNUODS4QLMUFERBGEQE5TEQ, seg_pos={0 0}, upload=446377]: Done uploading piece. err: <nil> Jul 16 15:21:38.560852907 (558h11m13.331494242s) [noise=true , address=45.91.95.3:4036 , piece_id=6WGBB66MJWJ2GBWKQROMZBDLLRHEPBOA5HDRXZIOPIU226ENBJEA, seg_pos={0 0}, upload=446377]: Done uploading piece. err: <nil> Jul 16 15:21:38.587805277 (558h11m13.358446612s) [noise=true , address=156.146.43.225:4457 , piece_id=IMMKGRXALKFMWNPVB7U3XKQOQFR5USOOJZ4DHQV3GSXAVVGP64FQ, seg_pos={0 0}, upload=446377]: Done uploading piece. err: <nil> Jul 16 15:21:38.597713243 (558h11m13.368354578s) [noise=true , address=45.91.116.8:4078 , piece_id=4Q7U5S4YMPXZ3APP5JYF2GXFDC3ZJUQ6HJULDNKUHJ7OUB364SPA, seg_pos={0 0}, upload=446377]: Done uploading piece. err: <nil> Jul 16 15:21:38.599006327 (558h11m13.369647662s) [noise=true , address=45.91.116.13:4159 , piece_id=5B34MTGYOC3BYPDKVMNSQNTYSWZWB56FEPES63E6RJRQ7HIL6GCA, seg_pos={0 0}, upload=446377]: Done uploading piece. err: <nil> Jul 16 15:21:38.682044158 (558h11m13.452685493s) [noise=true , address=79.127.219.48:4084 , piece_id=OUD2PZT6H5I5SDTBWAMW6UYNQI4GJQ3WFWO5UNGDB4G7ILVQQADA, seg_pos={0 0}, upload=446377]: Done uploading piece. err: <nil> Jul 16 15:21:38.710856095 (558h11m13.48149743s) [noise=true , address=104.36.82.8:4983 , piece_id=4LT7B7JH7KLR43PEOAS4FBCM7GARNRRHGBFXVNOQGDYGQG3ZZUZA, seg_pos={0 0}, upload=446377]: Done uploading piece. err: <nil> Jul 16 15:21:38.724282394 (558h11m13.494923729s) [noise=true , address=104.36.82.12:4154 , piece_id=LN56DTNFBMOGDH3NEEPLNNU65ZZ4DKF3ZNCRHIRODPHGFYGN5PMA, seg_pos={0 0}, upload=446377]: Done uploading piece. err: <nil> Jul 16 15:21:38.736872973 (558h11m13.507514359s) [noise=true , address=104.36.82.18:4386 , piece_id=6ZHILKLTD74MZA2M6ZM5R2MDW2BAD3GN7EZOXNI6LXLY7ZVSY2SA, seg_pos={0 0}, upload=446377]: Done uploading piece. err: <nil> Jul 16 15:21:38.747087357 (558h11m13.517728692s) [noise=true , address=45.91.116.11:4142 , piece_id=QQGR5PIFCHPRWBUVMVTUKAHAISBLSKZNR6OCIR4NM7RXUJO7UX7A, seg_pos={0 0}, upload=446377]: Done uploading piece. err: <nil> Jul 16 15:21:38.780196437 (558h11m13.550837772s) [noise=true , address=104.36.82.51:4072 , piece_id=C3OHCFSFJ3B7WW4ZCIGNTOL3XVBHEVHFYW6OIJTZJY44MDTYD5AA, seg_pos={0 0}, upload=446377]: Done uploading piece. err: <nil> Jul 16 15:21:38.807244329 (558h11m13.577885664s) [noise=true , address=45.91.116.9:4293 , piece_id=BHXNYDU3FMQE5ABEPKHNSHB6MXW6KEKTPTGEUPR7JRE2EERQH6OQ, seg_pos={0 0}, upload=446377]: Done uploading piece. err: <nil> Jul 16 15:22:35.602262422 (558h12m10.372903757s) [noise=true , address=77.90.31.10:4538 , piece_id=O5776XFIWA75ESMGPMJXUV37RLOG2FPW3CFYR6ZCEQKOM6XQ57LA, seg_pos={0 0}, upload=446377]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:22:35.602318459 (558h12m10.372959794s) [noise=true , address=77.90.31.11:4556 , piece_id=IWX3HP7PMMEZJH2BFZM6UQMBS75CCIOZZJKXZS3MYJCY54C4MMHA, seg_pos={0 0}, upload=446377]: Done uploading piece. err: ecclient: upload failed (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, address:77.90.31.11:4556): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:22:35.602534603 (558h12m10.373175938s) [noise=true , address=77.90.31.19:4851 , piece_id=4OHND2X7VSZ7W25MW63MTNY6ID2I4YHULDZQIQ45ZG5YPBRZO6HA, seg_pos={0 0}, upload=446377]: Done uploading piece. err: ecclient: upload failed (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk, address:77.90.31.19:4851): protocol: write tcp 188.166.225.170:39300->77.90.31.19:4851: use of closed network connection; write tcp 188.166.225.170:39300->77.90.31.19:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39300->77.90.31.19:4851: use of closed network connection Jul 16 15:22:35.602591101 (558h12m10.373232436s) [noise=true , address=77.90.31.15:4727 , piece_id=RHF2PKY3VG7G7RE4ML3YYCZSUGMLDWWYNODHZOK5AHGTEM2QLPOA, seg_pos={0 0}, upload=446377]: Done uploading piece. err: ecclient: upload failed (node:1xoAbv6zJuG9LjyCdsCrTWQaCDyLrWyv8RCrMTiqNqazoEwD1M, address:77.90.31.15:4727): protocol: write tcp 188.166.225.170:39106->77.90.31.15:4727: use of closed network connection; write tcp 188.166.225.170:39106->77.90.31.15:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39106->77.90.31.15:4727: use of closed network connection Jul 16 15:22:35.602678709 (558h12m10.373320034s) [noise=true , address=77.90.31.21:4894 , piece_id=U5CIB5APFQDRGRR2QPDGSAXXWUOYAVT4W3MK2ON4LIQSSHDDXO2A, seg_pos={0 0}, upload=446377]: Done uploading piece. err: ecclient: upload failed (node:1JJ81nDtVEinZaEU7NM7AAhCWiFN7V6a8o5Cp9ZzWjvjDCKmMG, address:77.90.31.21:4894): protocol: context deadline exceeded; context deadline exceeded Jul 16 15:22:35.602693527 (558h12m10.373334862s) [seg_pos={0 0}, upload=446377]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 16 15:22:35.835976116 (558h12m10.606617451s) [upload=446377]: Done uploading segments... Jul 16 15:22:35.836002216 (558h12m10.606643551s) [upload=446377]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.869769981s core uplink us-select-1 us1-placement-20
checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 900.076881ms remote: "tcp:5.161.37.179:443" response headers: Content-Length=0 Date=Wed, 16 Jul 2025 15:03:45 GMT Access-Control-Allow-Origin=* target: https://www.storj-ipfs.com/api/v0/add FAIL (2025-07-16 15:22:53 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "" Duration: 900.105205ms edge ipfs us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.235363081s FAIL (2025-07-16 15:22:35 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 11.015974179s) [upload=446377]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:21:36.245338395 (558h11m11.01597973s) [upload=446377]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:21:36.690946493 (558h11m11.461587828s) [seg_pos={0 0}, upload=446377]: Begin upload segment... Jul 16 15:21:36.690972012 (558h11m11.461613337s) [seg_pos={0 0}, upload=446377]: Done begin upload segment. Jul 16 15:21:36.690986640 (558h11m11.461627966s) [noise=true , address=104.36.82.18:4386 , piece_id=6ZHILKLTD74MZA2M6ZM5R2MDW2BAD3GN7EZOXNI6LXLY7ZVSY2SA, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.691041756 (558h11m11.461683091s) [noise=true , address=104.36.82.12:4154 , piece_id=LN56DTNFBMOGDH3NEEPLNNU65ZZ4DKF3ZNCRHIRODPHGFYGN5PMA, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.691307764 (558h11m11.461949089s) [noise=true , address=45.91.116.13:4159 , piece_id=5B34MTGYOC3BYPDKVMNSQNTYSWZWB56FEPES63E6RJRQ7HIL6GCA, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.691424208 (558h11m11.462065533s) [noise=true , address=45.91.95.3:4036 , piece_id=6WGBB66MJWJ2GBWKQROMZBDLLRHEPBOA5HDRXZIOPIU226ENBJEA, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.691568764 (558h11m11.462210099s) [noise=true , address=45.91.95.5:4013 , piece_id=SJ7TKNQ2G642VVORSNSPL7AV7BUV2JNUODS4QLMUFERBGEQE5TEQ, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.691718871 (558h11m11.462360196s) [noise=true , address=156.146.43.225:4457 , piece_id=IMMKGRXALKFMWNPVB7U3XKQOQFR5USOOJZ4DHQV3GSXAVVGP64FQ, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.691837559 (558h11m11.462478894s) [noise=true , address=77.90.31.19:4851 , piece_id=4OHND2X7VSZ7W25MW63MTNY6ID2I4YHULDZQIQ45ZG5YPBRZO6HA, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.692008656 (558h11m11.462649991s) [noise=true , address=77.90.31.11:4556 , piece_id=IWX3HP7PMMEZJH2BFZM6UQMBS75CCIOZZJKXZS3MYJCY54C4MMHA, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.692094189 (558h11m11.462735524s) [noise=true , address=79.127.219.48:4084 , piece_id=OUD2PZT6H5I5SDTBWAMW6UYNQI4GJQ3WFWO5UNGDB4G7ILVQQADA, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.692290194 (558h11m11.462931529s) [noise=true , address=77.90.31.10:4538 , piece_id=O5776XFIWA75ESMGPMJXUV37RLOG2FPW3CFYR6ZCEQKOM6XQ57LA, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.692332916 (558h11m11.462974241s) [noise=true , address=45.91.116.8:4078 , piece_id=4Q7U5S4YMPXZ3APP5JYF2GXFDC3ZJUQ6HJULDNKUHJ7OUB364SPA, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.692537698 (558h11m11.463179033s) [noise=true , address=45.91.116.9:4293 , piece_id=BHXNYDU3FMQE5ABEPKHNSHB6MXW6KEKTPTGEUPR7JRE2EERQH6OQ, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.692542156 (558h11m11.463183491s) [noise=true , address=77.90.31.15:4727 , piece_id=RHF2PKY3VG7G7RE4ML3YYCZSUGMLDWWYNODHZOK5AHGTEM2QLPOA, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.692741288 (558h11m11.463382613s) [noise=true , address=104.36.82.51:4072 , piece_id=C3OHCFSFJ3B7WW4ZCIGNTOL3XVBHEVHFYW6OIJTZJY44MDTYD5AA, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.692785722 (558h11m11.463427047s) [noise=true , address=77.90.31.21:4894 , piece_id=U5CIB5APFQDRGRR2QPDGSAXXWUOYAVT4W3MK2ON4LIQSSHDDXO2A, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.692982458 (558h11m11.463623794s) [noise=true , address=45.91.116.11:4142 , piece_id=QQGR5PIFCHPRWBUVMVTUKAHAISBLSKZNR6OCIR4NM7RXUJO7UX7A, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.693070959 (558h11m11.463712284s) [noise=true , address=104.36.82.8:4983 , piece_id=4LT7B7JH7KLR43PEOAS4FBCM7GARNRRHGBFXVNOQGDYGQG3ZZUZA, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:36.694049761 (558h11m11.464691086s) [noise=true , address=45.91.95.2:4011 , piece_id=3IICDUFHRFGRSLUUIB6CZJAU67S2X6FRU63U2MBDUKEFZP4C4HGQ, seg_pos={0 0}, upload=446377]: Uploading piece... Jul 16 15:21:38.008052223 (558h11m12.778693558s) [upload=446377]: Next returned nil segment Jul 16 15:21:38.008070457 (558h11m12.778711792s) [upload=446377]: Waiting for error group managing segments... Jul 16 15:21:38.370664362 (558h11m13.141305697s) [noise=true , address=45.91.95.2:4011 , piece_id=3IICDUFHRFGRSLUUIB6CZJAU67S2X6FRU63U2MBDUKEFZP4C4HGQ, seg_pos={0 0}, upload=446377]: Done uploading piece. err: <nil> Jul 16 15:21:38.394746470 (558h11m13.165387805s) [noise=true , address=45.91.95.5:4013 , piece_id=SJ7TKNQ2G642VVORSNSPL7AV7BUV2JNUODS4QLMUFERBGEQE5TEQ, seg_pos={0 0}, upload=446377]: Done uploading piece. err: <nil> Jul 16 15:21:38.560852907 (558h11m13.331494242s) [noise=true , address=45.91.95.3:4036 , piece_id=6WGBB66MJWJ2GBWKQROMZBDLLRHEPBOA5HDRXZIOPIU226ENBJEA, seg_pos={0 0}, upload=446377]: Done uploading piece. err: <nil> Jul 16 15:21:38.587805277 (558h11m13.358446612s) [noise=true , address=156.146.43.225:4457 , piece_id=IMMKGRXALKFMWNPVB7U3XKQOQFR5USOOJZ4DHQV3GSXAVVGP64FQ, seg_pos={0 0}, upload=446377]: Done uploading piece. err: <nil> Jul 16 15:21:38.597713243 (558h11m13.368354578s) [noise=true , address=45.91.116.8:4078 , piece_id=4Q7U5S4YMPXZ3APP5JYF2GXFDC3ZJUQ6HJULDNKUHJ7OUB364SPA, seg_pos={0 0}, upload=446377]: Done uploading piece. err: <nil> Jul 16 15:21:38.599006327 (558h11m13.369647662s) [noise=true , address=45.91.116.13:4159 , piece_id=5B34MTGYOC3BYPDKVMNSQNTYSWZWB56FEPES63E6RJRQ7HIL6GCA, seg_pos={0 0}, upload=446377]: Done uploading piece. err: <nil> Jul 16 15:21:38.682044158 (558h11m13.452685493s) [noise=true , address=79.127.219.48:4084 , piece_id=OUD2PZT6H5I5SDTBWAMW6UYNQI4GJQ3WFWO5UNGDB4G7ILVQQADA, seg_pos={0 0}, upload=446377]: Done uploading piece. err: <nil> Jul 16 15:21:38.710856095 (558h11m13.48149743s) [noise=true , address=104.36.82.8:4983 , piece_id=4LT7B7JH7KLR43PEOAS4FBCM7GARNRRHGBFXVNOQGDYGQG3ZZUZA, seg_pos={0 0}, upload=446377]: Done uploading piece. err: <nil> Jul 16 15:21:38.724282394 (558h11m13.494923729s) [noise=true , address=104.36.82.12:4154 , piece_id=LN56DTNFBMOGDH3NEEPLNNU65ZZ4DKF3ZNCRHIRODPHGFYGN5PMA, seg_pos={0 0}, upload=446377]: Done uploading piece. err: <nil> Jul 16 15:21:38.736872973 (558h11m13.507514359s) [noise=true , address=104.36.82.18:4386 , piece_id=6ZHILKLTD74MZA2M6ZM5R2MDW2BAD3GN7EZOXNI6LXLY7ZVSY2SA, seg_pos={0 0}, upload=446377]: Done uploading piece. err: <nil> Jul 16 15:21:38.747087357 (558h11m13.517728692s) [noise=true , address=45.91.116.11:4142 , piece_id=QQGR5PIFCHPRWBUVMVTUKAHAISBLSKZNR6OCIR4NM7RXUJO7UX7A, seg_pos={0 0}, upload=446377]: Done uploading piece. err: <nil> Jul 16 15:21:38.780196437 (558h11m13.550837772s) [noise=true , address=104.36.82.51:4072 , piece_id=C3OHCFSFJ3B7WW4ZCIGNTOL3XVBHEVHFYW6OIJTZJY44MDTYD5AA, seg_pos={0 0}, upload=446377]: Done uploading piece. err: <nil> Jul 16 15:21:38.807244329 (558h11m13.577885664s) [noise=true , address=45.91.116.9:4293 , piece_id=BHXNYDU3FMQE5ABEPKHNSHB6MXW6KEKTPTGEUPR7JRE2EERQH6OQ, seg_pos={0 0}, upload=446377]: Done uploading piece. err: <nil> Jul 16 15:22:35.602262422 (558h12m10.372903757s) [noise=true , address=77.90.31.10:4538 , piece_id=O5776XFIWA75ESMGPMJXUV37RLOG2FPW3CFYR6ZCEQKOM6XQ57LA, seg_pos={0 0}, upload=446377]: Done uploading piece. err: ecclient: upload failed (node:12kRwLj3eRSu1a8oRSnUF5NuG6sJ9R9qMmFT89CHXMWB11WQD1N, address:77.90.31.10:4538): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:22:35.602318459 (558h12m10.372959794s) [noise=true , address=77.90.31.11:4556 , piece_id=IWX3HP7PMMEZJH2BFZM6UQMBS75CCIOZZJKXZS3MYJCY54C4MMHA, seg_pos={0 0}, upload=446377]: Done uploading piece. err: ecclient: upload failed (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, address:77.90.31.11:4556): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:22:35.602534603 (558h12m10.373175938s) [noise=true , address=77.90.31.19:4851 , piece_id=4OHND2X7VSZ7W25MW63MTNY6ID2I4YHULDZQIQ45ZG5YPBRZO6HA, seg_pos={0 0}, upload=446377]: Done uploading piece. err: ecclient: upload failed (node:1NEV2ndHw4fohVUNsfrzwojd8S11TmMST6Sz59GVnXS8vDGhWk, address:77.90.31.19:4851): protocol: write tcp 188.166.225.170:39300->77.90.31.19:4851: use of closed network connection; write tcp 188.166.225.170:39300->77.90.31.19:4851: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39300->77.90.31.19:4851: use of closed network connection Jul 16 15:22:35.602591101 (558h12m10.373232436s) [noise=true , address=77.90.31.15:4727 , piece_id=RHF2PKY3VG7G7RE4ML3YYCZSUGMLDWWYNODHZOK5AHGTEM2QLPOA, seg_pos={0 0}, upload=446377]: Done uploading piece. err: ecclient: upload failed (node:1xoAbv6zJuG9LjyCdsCrTWQaCDyLrWyv8RCrMTiqNqazoEwD1M, address:77.90.31.15:4727): protocol: write tcp 188.166.225.170:39106->77.90.31.15:4727: use of closed network connection; write tcp 188.166.225.170:39106->77.90.31.15:4727: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39106->77.90.31.15:4727: use of closed network connection Jul 16 15:22:35.602678709 (558h12m10.373320034s) [noise=true , address=77.90.31.21:4894 , piece_id=U5CIB5APFQDRGRR2QPDGSAXXWUOYAVT4W3MK2ON4LIQSSHDDXO2A, seg_pos={0 0}, upload=446377]: Done uploading piece. err: ecclient: upload failed (node:1JJ81nDtVEinZaEU7NM7AAhCWiFN7V6a8o5Cp9ZzWjvjDCKmMG, address:77.90.31.21:4894): protocol: context deadline exceeded; context deadline exceeded Jul 16 15:22:35.602693527 (558h12m10.373334862s) [seg_pos={0 0}, upload=446377]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 16 15:22:35.835976116 (558h12m10.606617451s) [upload=446377]: Done uploading segments... Jul 16 15:22:35.836002216 (558h12m10.606643551s) [upload=446377]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.235378982s core uplink us-select-1 us1-placement-20
checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 891.835233ms remote: "tcp:5.161.37.179:443" response headers: Content-Length=0 Date=Wed, 16 Jul 2025 15:03:45 GMT Access-Control-Allow-Origin=* target: https://www.storj-ipfs.com/api/v0/add FAIL (2025-07-16 15:21:47 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "" Duration: 891.86035ms edge ipfs us1
checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 913.836129ms remote: "tcp:5.161.37.179:443" response headers: Content-Length=0 Date=Wed, 16 Jul 2025 15:03:45 GMT Access-Control-Allow-Origin=* target: https://www.storj-ipfs.com/api/v0/add FAIL (2025-07-16 15:20:52 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "" Duration: 913.862939ms edge ipfs us1
checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 870.912498ms remote: "tcp:5.161.37.179:443" response headers: Content-Length=0 Date=Wed, 16 Jul 2025 15:03:45 GMT Access-Control-Allow-Origin=* target: https://www.storj-ipfs.com/api/v0/add FAIL (2025-07-16 15:20:33 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "" Duration: 870.942896ms edge ipfs us1
checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 991.672935ms remote: "tcp:5.161.37.179:443" response headers: Content-Length=0 Date=Wed, 16 Jul 2025 15:03:45 GMT Access-Control-Allow-Origin=* target: https://www.storj-ipfs.com/api/v0/add FAIL (2025-07-16 15:20:08 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "" Duration: 991.699766ms edge ipfs us1
checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 1.120651466s remote: "tcp:5.161.37.179:443" response headers: Content-Length=0 Date=Wed, 16 Jul 2025 15:03:45 GMT Access-Control-Allow-Origin=* target: https://www.storj-ipfs.com/api/v0/add FAIL (2025-07-16 15:19:40 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "" Duration: 1.120679259s edge ipfs us1
checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 1.149664849s remote: "tcp:5.161.37.179:443" response headers: Content-Length=0 Date=Wed, 16 Jul 2025 15:03:45 GMT Access-Control-Allow-Origin=* target: https://www.storj-ipfs.com/api/v0/add FAIL (2025-07-16 15:19:06 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "" Duration: 1.149708281s edge ipfs us1
checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 919.155895ms remote: "tcp:5.161.37.179:443" response headers: Content-Length=0 Date=Wed, 16 Jul 2025 15:03:45 GMT Access-Control-Allow-Origin=* target: https://www.storj-ipfs.com/api/v0/add FAIL (2025-07-16 15:18:16 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "" Duration: 919.18546ms edge ipfs us1
checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 869.843522ms remote: "tcp:5.161.37.179:443" response headers: Content-Length=0 Date=Wed, 16 Jul 2025 15:03:45 GMT Access-Control-Allow-Origin=* target: https://www.storj-ipfs.com/api/v0/add FAIL (2025-07-16 15:17:50 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "" Duration: 869.871175ms edge ipfs us1
checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 861.913238ms remote: "tcp:5.161.37.179:443" response headers: Content-Length=0 Date=Wed, 16 Jul 2025 15:03:45 GMT Access-Control-Allow-Origin=* target: https://www.storj-ipfs.com/api/v0/add FAIL (2025-07-16 15:16:40 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "" Duration: 861.939428ms edge ipfs us1
checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 883.219927ms remote: "tcp:5.161.37.179:443" response headers: Content-Length=0 Date=Wed, 16 Jul 2025 15:03:45 GMT Access-Control-Allow-Origin=* target: https://www.storj-ipfs.com/api/v0/add FAIL (2025-07-16 15:16:22 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "" Duration: 883.236289ms edge ipfs us1
checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 916.768216ms remote: "tcp:5.161.37.179:443" response headers: Content-Length=0 Date=Wed, 16 Jul 2025 15:03:45 GMT Access-Control-Allow-Origin=* target: https://www.storj-ipfs.com/api/v0/add FAIL (2025-07-16 15:15:11 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "" Duration: 916.791811ms edge ipfs us1
checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 906.69457ms remote: "tcp:5.161.37.179:443" response headers: Content-Length=0 Date=Wed, 16 Jul 2025 15:03:45 GMT Access-Control-Allow-Origin=* target: https://www.storj-ipfs.com/api/v0/add FAIL (2025-07-16 15:14:13 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "" Duration: 906.718957ms edge ipfs us1
checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 1.216221169s remote: "tcp:5.161.37.179:443" response headers: Content-Length=0 Date=Wed, 16 Jul 2025 15:03:45 GMT Access-Control-Allow-Origin=* target: https://www.storj-ipfs.com/api/v0/add FAIL (2025-07-16 15:13:19 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "" Duration: 1.216239223s edge ipfs us1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post)

checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post)

checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post)

checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post)

checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post)

checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post)

checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post)

checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post)

checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post)

checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post)

checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post)

checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post)

checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post)

checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post)

checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post)

checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post)