Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.309774149s FAIL (2025-07-16 16:07:35 UTC) orbiter warning: context deadline exceeded Duration: 6.30979595s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.241584663s FAIL (2025-07-16 16:07:20 UTC) orbiter warning: context deadline exceeded Duration: 6.241608698s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [....X..XX...................................................] duration: 5.670346667s FAIL (2025-07-16 16:07:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.670370723s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......XXXX.................................................] duration: 5.786229228s FAIL (2025-07-16 16:07:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.786260008s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 6.2864431s FAIL (2025-07-16 16:06:27 UTC) orbiter warning: context deadline exceeded Duration: 6.286483766s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.....X..XX..................................................] duration: 5.582025999s FAIL (2025-07-16 16:06:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.582054453s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........XXXX................................................] duration: 6.030351259s FAIL (2025-07-16 16:06:05 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.030381599s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........XXXX...............................................] duration: 5.856474131s FAIL (2025-07-16 16:05:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.856499679s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.104783254s FAIL (2025-07-16 16:05:20 UTC) orbiter warning: context deadline exceeded Duration: 6.104806226s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [......X..XX.................................................] duration: 5.987760449s FAIL (2025-07-16 16:05:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.987788684s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........XXXX..............................................] duration: 5.999332152s FAIL (2025-07-16 16:04:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.999360556s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 6.202567689s FAIL (2025-07-16 16:04:27 UTC) orbiter warning: context deadline exceeded Duration: 6.202588929s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.......X..XX................................................] duration: 5.789459774s FAIL (2025-07-16 16:04:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.789488169s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [........X..XX...............................................] duration: 5.851804319s FAIL (2025-07-16 16:03:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.851820801s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 6.295221936s FAIL (2025-07-16 16:03:57 UTC) orbiter warning: context deadline exceeded Duration: 6.2952344s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........XXXX.............................................] duration: 5.689461856s FAIL (2025-07-16 16:03:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.689488216s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............XXXX............................................] duration: 5.686478285s FAIL (2025-07-16 16:03:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.686498284s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.........X..XX..............................................] duration: 11.618838251s FAIL (2025-07-16 16:03:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 11.618866476s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 6.259123878s FAIL (2025-07-16 16:02:46 UTC) orbiter warning: context deadline exceeded Duration: 6.259147243s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............XXXX...........................................] duration: 6.161682259s FAIL (2025-07-16 16:02:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.161710243s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..........X..XX.............................................] duration: 5.723882552s FAIL (2025-07-16 16:02:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.723909492s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.265986841s FAIL (2025-07-16 16:01:46 UTC) orbiter warning: context deadline exceeded Duration: 6.266008534s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...........X..XX............................................] duration: 5.759841828s FAIL (2025-07-16 16:01:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.75986881s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............XXXX..........................................] duration: 5.889496761s FAIL (2025-07-16 16:01:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.889524482s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.502067926s FAIL (2025-07-16 16:01:10 UTC) orbiter warning: context deadline exceeded Duration: 6.502092422s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [............X..XX...........................................] duration: 6.423756837s FAIL (2025-07-16 16:01:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 6.423782497s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............XXXX.........................................] duration: 15.991761008s FAIL (2025-07-16 16:01:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 15.991790763s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................XXXX........................................] duration: 5.989516846s FAIL (2025-07-16 16:00:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.989546891s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.............X..XX..........................................] duration: 6.232796512s FAIL (2025-07-16 16:00:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 6.232827731s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 6.340097697s FAIL (2025-07-16 16:00:01 UTC) orbiter warning: context deadline exceeded Duration: 6.340119597s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.157036272s FAIL (2025-07-16 15:59:40 UTC) orbiter warning: context deadline exceeded Duration: 6.157075347s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................XXXX.......................................] duration: 5.775894036s FAIL (2025-07-16 15:59:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.775919644s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..............X..XX.........................................] duration: 5.779331061s FAIL (2025-07-16 15:59:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.779359264s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.294316897s FAIL (2025-07-16 15:58:36 UTC) orbiter warning: context deadline exceeded Duration: 6.294339911s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...............X..XX........................................] duration: 5.45416017s FAIL (2025-07-16 15:58:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.454190028s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 6.113196111s FAIL (2025-07-16 15:57:59 UTC) orbiter warning: context deadline exceeded Duration: 6.113273939s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................XXXX......................................] duration: 5.833911953s FAIL (2025-07-16 15:57:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.833939847s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................XXXX.....................................] duration: 5.644553205s FAIL (2025-07-16 15:57:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.644575155s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.81170668s FAIL (2025-07-16 15:57:26 UTC) orbiter warning: context deadline exceeded Duration: 6.811731439s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [................X..XX.......................................] duration: 5.637630863s FAIL (2025-07-16 15:57:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.637658766s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................XXXX....................................] duration: 5.737522046s FAIL (2025-07-16 15:57:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.737545519s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.................X..XX......................................] duration: 5.700585719s FAIL (2025-07-16 15:56:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.700613803s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.416549279s FAIL (2025-07-16 15:56:41 UTC) orbiter warning: context deadline exceeded Duration: 6.416578855s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................XXXX...................................] duration: 5.713801428s FAIL (2025-07-16 15:56:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.71383302s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..................X..XX.....................................] duration: 5.78160338s FAIL (2025-07-16 15:56:05 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.781630562s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................XXXX..................................] duration: 5.827762279s FAIL (2025-07-16 15:55:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.82779418s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 6.056186194s FAIL (2025-07-16 15:55:30 UTC) orbiter warning: context deadline exceeded Duration: 6.0562046s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...................X..XX....................................] duration: 5.89814261s FAIL (2025-07-16 15:55:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.898167097s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [....................X..XX...................................] duration: 5.56205372s FAIL (2025-07-16 15:54:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.562085711s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 6.222902368s FAIL (2025-07-16 15:54:37 UTC) orbiter warning: context deadline exceeded Duration: 6.222924722s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)