Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [........................X..XX...............................] duration: 5.528549166s FAIL (2025-07-16 15:51:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.528575837s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.326341526s FAIL (2025-07-16 15:50:42 UTC) orbiter warning: context deadline exceeded Duration: 6.326363819s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.........................X..XX..............................] duration: 5.728773591s FAIL (2025-07-16 15:50:10 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.728809199s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............................XXXX............................] duration: 5.799507799s FAIL (2025-07-16 15:50:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.799542805s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 12.253447866s FAIL (2025-07-16 15:49:48 UTC) orbiter warning: context deadline exceeded Duration: 12.25347153s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............................XXXX...........................] duration: 5.873132097s FAIL (2025-07-16 15:49:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.873158037s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..........................X..XX.............................] duration: 5.62863271s FAIL (2025-07-16 15:49:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.628660553s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 6.396572875s FAIL (2025-07-16 15:48:51 UTC) orbiter warning: context deadline exceeded Duration: 6.396593204s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............................XXXX..........................] duration: 5.856797034s FAIL (2025-07-16 15:48:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.856814547s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...........................X..XX............................] duration: 5.569126561s FAIL (2025-07-16 15:48:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.569159253s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 7.501096129s FAIL (2025-07-16 15:47:59 UTC) orbiter warning: context deadline exceeded Duration: 7.501179488s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............................XXXX.........................] duration: 5.75570911s FAIL (2025-07-16 15:47:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.755732333s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [............................X..XX...........................] duration: 5.548025108s FAIL (2025-07-16 15:47:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.548052128s core uplink us-select-1 us1-placement-20
checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 849.049685ms remote: "tcp:5.161.37.179:443" response headers: Content-Length=0 Date=Wed, 16 Jul 2025 15:03:45 GMT Access-Control-Allow-Origin=* target: https://www.storj-ipfs.com/api/v0/add FAIL (2025-07-16 15:47:16 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "" Duration: 849.072108ms edge ipfs us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................................XXXX........................] duration: 5.779257764s FAIL (2025-07-16 15:46:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.779279567s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.............................X..XX..........................] duration: 5.751530292s FAIL (2025-07-16 15:46:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.751558898s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.387425764s FAIL (2025-07-16 15:46:47 UTC) orbiter warning: context deadline exceeded Duration: 6.387443087s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 14.218269034s FAIL (2025-07-16 15:46:34 UTC) orbiter warning: context deadline exceeded Duration: 14.218288972s edge gateway us-select-1
checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 935.363142ms remote: "tcp:5.161.37.179:443" response headers: Content-Length=0 Date=Wed, 16 Jul 2025 15:03:45 GMT Access-Control-Allow-Origin=* target: https://www.storj-ipfs.com/api/v0/add FAIL (2025-07-16 15:46:19 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "" Duration: 935.397328ms edge ipfs us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................................XXXX.......................] duration: 5.693982133s FAIL (2025-07-16 15:46:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.694010518s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.730255491s FAIL (2025-07-16 15:46:00 UTC) orbiter warning: context deadline exceeded Duration: 6.730282403s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..............................X..XX.........................] duration: 5.782776347s FAIL (2025-07-16 15:45:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.782801566s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...............................X..XX........................] duration: 5.503452679s FAIL (2025-07-16 15:45:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.503479681s core uplink us-select-1 us1-placement-20
checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 906.741549ms remote: "tcp:5.161.37.179:443" response headers: Content-Length=0 Date=Wed, 16 Jul 2025 15:03:45 GMT Access-Control-Allow-Origin=* target: https://www.storj-ipfs.com/api/v0/add FAIL (2025-07-16 15:45:29 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "" Duration: 906.769152ms edge ipfs us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................................XXXX......................] duration: 5.740784962s FAIL (2025-07-16 15:45:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.740855477s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 6.078701075s FAIL (2025-07-16 15:45:14 UTC) orbiter warning: context deadline exceeded Duration: 6.078723668s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................................XXXX.....................] duration: 5.788966241s FAIL (2025-07-16 15:44:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.788993192s core uplink us-select-1
checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 918.721911ms remote: "tcp:5.161.37.179:443" response headers: Content-Length=0 Date=Wed, 16 Jul 2025 15:03:45 GMT Access-Control-Allow-Origin=* target: https://www.storj-ipfs.com/api/v0/add FAIL (2025-07-16 15:44:25 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "" Duration: 918.745157ms edge ipfs us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................................XXXX....................] duration: 5.815293674s FAIL (2025-07-16 15:44:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.815319804s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.230678514s FAIL (2025-07-16 15:44:24 UTC) orbiter warning: context deadline exceeded Duration: 6.230702621s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [................................X..XX.......................] duration: 5.836712762s FAIL (2025-07-16 15:44:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.836737791s core uplink us-select-1 us1-placement-20
checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 851.552057ms remote: "tcp:5.161.37.179:443" response headers: Content-Length=0 Date=Wed, 16 Jul 2025 15:03:45 GMT Access-Control-Allow-Origin=* target: https://www.storj-ipfs.com/api/v0/add FAIL (2025-07-16 15:43:31 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "" Duration: 851.593496ms edge ipfs us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.971617629s FAIL (2025-07-16 15:43:23 UTC) orbiter warning: context deadline exceeded Duration: 6.971656734s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.................................X..XX......................] duration: 5.808211694s FAIL (2025-07-16 15:43:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.808236792s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................................XXXX...................] duration: 5.685962679s FAIL (2025-07-16 15:43:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.685992276s core uplink us-select-1
checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 931.968416ms remote: "tcp:5.161.37.179:443" response headers: Content-Length=0 Date=Wed, 16 Jul 2025 15:03:45 GMT Access-Control-Allow-Origin=* target: https://www.storj-ipfs.com/api/v0/add FAIL (2025-07-16 15:43:00 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "" Duration: 931.987443ms edge ipfs us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................................XXXX..................] duration: 5.995880246s FAIL (2025-07-16 15:42:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.995905985s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 7.424677142s FAIL (2025-07-16 15:42:43 UTC) orbiter warning: context deadline exceeded Duration: 7.424694387s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..................................X..XX.....................] duration: 5.839336828s FAIL (2025-07-16 15:42:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.839363389s core uplink us-select-1 us1-placement-20
checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 876.889645ms remote: "tcp:5.161.37.179:443" response headers: Content-Length=0 Date=Wed, 16 Jul 2025 15:03:45 GMT Access-Control-Allow-Origin=* target: https://www.storj-ipfs.com/api/v0/add FAIL (2025-07-16 15:41:59 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "" Duration: 876.906016ms edge ipfs us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.182166086s FAIL (2025-07-16 15:41:53 UTC) orbiter warning: context deadline exceeded Duration: 6.182189279s edge gateway us-select-1
checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 962.342605ms remote: "tcp:5.161.37.179:443" response headers: Content-Length=0 Date=Wed, 16 Jul 2025 15:03:45 GMT Access-Control-Allow-Origin=* target: https://www.storj-ipfs.com/api/v0/add FAIL (2025-07-16 15:41:39 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "" Duration: 962.365048ms edge ipfs us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................................XXXX.................] duration: 5.691437506s FAIL (2025-07-16 15:41:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.69147135s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 12.352786857s FAIL (2025-07-16 15:41:15 UTC) orbiter warning: context deadline exceeded Duration: 12.35280862s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...................................X..XX....................] duration: 5.753927663s FAIL (2025-07-16 15:41:13 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.753948051s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................................XXXX................] duration: 5.769183079s FAIL (2025-07-16 15:40:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.769209499s core uplink us-select-1
checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 905.956878ms remote: "tcp:5.161.37.179:443" response headers: Content-Length=0 Date=Wed, 16 Jul 2025 15:03:45 GMT Access-Control-Allow-Origin=* target: https://www.storj-ipfs.com/api/v0/add FAIL (2025-07-16 15:40:51 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "" Duration: 905.980243ms edge ipfs us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 11.12595872s FAIL (2025-07-16 15:40:36 UTC) orbiter warning: context deadline exceeded Duration: 11.126067149s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [....................................X..XX...................] duration: 5.669800829s FAIL (2025-07-16 15:40:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.669827489s core uplink us-select-1 us1-placement-20
checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 903.102468ms remote: "tcp:5.161.37.179:443" response headers: Content-Length=0 Date=Wed, 16 Jul 2025 15:03:45 GMT Access-Control-Allow-Origin=* target: https://www.storj-ipfs.com/api/v0/add FAIL (2025-07-16 15:40:30 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "" Duration: 903.127156ms edge ipfs us1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post)

checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.HTTP{"POST" "https://www.storj-ipfs.com/api/v0/add", authorization: <redacted>, ExpectedContent: "Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi"} failure (antiblip window 3/60): (ipfs.post)