Orbiter History

checkers.HTTP{"GET" "https://www.storj-ipfs.com/ipfs/Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi", ExpectedContent: "Storj IPFS!"} failure (antiblip window 3/60): (ipfs.get) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 1.152931351s remote: "tcp:5.161.37.179:443" response headers: Access-Control-Allow-Origin=* Date=Wed, 16 Jul 2025 16:28:03 GMT Content-Length=11 Content-Type=text/plain; charset=utf-8 target: https://www.storj-ipfs.com/ipfs/Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi FAIL (2025-07-16 16:29:19 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "Bad Gateway" Duration: 1.152952341s edge ipfs us1
checkers.HTTP{"GET" "https://www.storj-ipfs.com/ipfs/Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi", ExpectedContent: "Storj IPFS!"} failure (antiblip window 3/60): (ipfs.get) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 1.021432301s remote: "tcp:5.161.37.179:443" response headers: Access-Control-Allow-Origin=* Date=Wed, 16 Jul 2025 16:28:03 GMT Content-Length=11 Content-Type=text/plain; charset=utf-8 target: https://www.storj-ipfs.com/ipfs/Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi FAIL (2025-07-16 16:29:00 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "Bad Gateway" Duration: 1.021475864s edge ipfs us1
checkers.HTTP{"GET" "https://www.storj-ipfs.com/ipfs/Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi", ExpectedContent: "Storj IPFS!"} failure (antiblip window 3/60): (ipfs.get) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 710.319039ms remote: "tcp:5.161.37.179:443" response headers: Access-Control-Allow-Origin=* Date=Wed, 16 Jul 2025 16:28:03 GMT Content-Length=11 Content-Type=text/plain; charset=utf-8 target: https://www.storj-ipfs.com/ipfs/Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi FAIL (2025-07-16 16:28:03 UTC) orbiter warning: unexpected status code "502 Bad Gateway" "Bad Gateway" Duration: 710.342153ms edge ipfs us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.476233529s FAIL (2025-07-16 16:16:05 UTC) orbiter warning: context deadline exceeded Duration: 6.476260941s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.076589776s FAIL (2025-07-16 16:15:12 UTC) orbiter warning: context deadline exceeded Duration: 6.076611979s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.984788729s FAIL (2025-07-16 16:14:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.984813528s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.646920063s FAIL (2025-07-16 16:14:04 UTC) orbiter warning: context deadline exceeded Duration: 6.646946524s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 5.753805675s FAIL (2025-07-16 16:13:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.753826975s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.573314444s FAIL (2025-07-16 16:13:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.573340514s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 5.612091509s FAIL (2025-07-16 16:13:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.612114353s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.057295644s FAIL (2025-07-16 16:13:21 UTC) orbiter warning: context deadline exceeded Duration: 6.057347402s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 6.241814818s FAIL (2025-07-16 16:13:02 UTC) orbiter warning: context deadline exceeded Duration: 6.241837151s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [XXX.........................................................] duration: 5.815449993s FAIL (2025-07-16 16:12:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.815545396s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.XX.........................................................] duration: 5.69630426s FAIL (2025-07-16 16:12:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.696328067s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [XXXX........................................................] duration: 5.783661641s FAIL (2025-07-16 16:12:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.783689002s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.029809044s FAIL (2025-07-16 16:12:21 UTC) orbiter warning: context deadline exceeded Duration: 6.030015299s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..XX........................................................] duration: 5.603379578s FAIL (2025-07-16 16:11:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.603415527s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.XXXX.......................................................] duration: 5.755898771s FAIL (2025-07-16 16:11:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.755928669s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 6.548182148s FAIL (2025-07-16 16:11:07 UTC) orbiter warning: context deadline exceeded Duration: 6.548200585s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..XXXX......................................................] duration: 5.817224357s FAIL (2025-07-16 16:10:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.817255927s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [X..XX.......................................................] duration: 5.782137954s FAIL (2025-07-16 16:10:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.782169867s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...XXXX.....................................................] duration: 5.743853953s FAIL (2025-07-16 16:10:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.743893379s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....XXXX....................................................] duration: 5.991712136s FAIL (2025-07-16 16:10:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.991742093s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.X..XX......................................................] duration: 5.818076036s FAIL (2025-07-16 16:10:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.818104339s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 6.426511502s FAIL (2025-07-16 16:09:58 UTC) orbiter warning: context deadline exceeded Duration: 6.426544736s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..X..XX.....................................................] duration: 5.814631538s FAIL (2025-07-16 16:09:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.814657086s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....XXXX...................................................] duration: 5.822358006s FAIL (2025-07-16 16:09:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.82238112s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.468131186s FAIL (2025-07-16 16:09:13 UTC) orbiter warning: context deadline exceeded Duration: 6.468163969s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 6.071201242s FAIL (2025-07-16 16:08:52 UTC) orbiter warning: context deadline exceeded Duration: 6.071224174s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.838232432s FAIL (2025-07-16 16:08:37 UTC) orbiter warning: context deadline exceeded Duration: 6.838256358s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...X..XX....................................................] duration: 5.793034303s FAIL (2025-07-16 16:08:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.793061976s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.339334501s FAIL (2025-07-16 16:08:21 UTC) orbiter warning: context deadline exceeded Duration: 6.339357666s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......XXXX..................................................] duration: 5.912534169s FAIL (2025-07-16 16:08:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.912565918s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.309774149s FAIL (2025-07-16 16:07:35 UTC) orbiter warning: context deadline exceeded Duration: 6.30979595s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.241584663s FAIL (2025-07-16 16:07:20 UTC) orbiter warning: context deadline exceeded Duration: 6.241608698s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [....X..XX...................................................] duration: 5.670346667s FAIL (2025-07-16 16:07:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.670370723s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......XXXX.................................................] duration: 5.786229228s FAIL (2025-07-16 16:07:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.786260008s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 6.2864431s FAIL (2025-07-16 16:06:27 UTC) orbiter warning: context deadline exceeded Duration: 6.286483766s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.....X..XX..................................................] duration: 5.582025999s FAIL (2025-07-16 16:06:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.582054453s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........XXXX................................................] duration: 6.030351259s FAIL (2025-07-16 16:06:05 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.030381599s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........XXXX...............................................] duration: 5.856474131s FAIL (2025-07-16 16:05:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.856499679s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.104783254s FAIL (2025-07-16 16:05:20 UTC) orbiter warning: context deadline exceeded Duration: 6.104806226s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [......X..XX.................................................] duration: 5.987760449s FAIL (2025-07-16 16:05:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.987788684s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........XXXX..............................................] duration: 5.999332152s FAIL (2025-07-16 16:04:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.999360556s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 6.202567689s FAIL (2025-07-16 16:04:27 UTC) orbiter warning: context deadline exceeded Duration: 6.202588929s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.......X..XX................................................] duration: 5.789459774s FAIL (2025-07-16 16:04:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.789488169s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [........X..XX...............................................] duration: 5.851804319s FAIL (2025-07-16 16:03:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018012614 (558h15m1.788653949s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018419914 (558h15m1.789061249s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018724186 (558h15m1.789365521s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.018988672 (558h15m1.789630007s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019735040 (558h15m1.790376375s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.019946696 (558h15m1.790588031s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020156907 (558h15m1.790798242s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020392709 (558h15m1.791034044s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020593302 (558h15m1.791234637s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020783987 (558h15m1.791425312s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.020932071 (558h15m1.791573406s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021140639 (558h15m1.791781975s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021341625 (558h15m1.79198296s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021529133 (558h15m1.792170468s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.021728184 (558h15m1.792369519s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022122729 (558h15m1.792764064s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:27.022321359 (558h15m1.792962694s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Uploading piece... Jul 16 15:25:28.127893815 (558h15m2.89853514s) [upload=446427]: Next returned nil segment Jul 16 15:25:28.128245288 (558h15m2.898886613s) [upload=446427]: Waiting for error group managing segments... Jul 16 15:25:28.441686517 (558h15m3.212327842s) [noise=true , address=79.127.205.246:4256 , piece_id=HUZPR24DYSE377U42UQYHOLJR5R3MFDYC2CANXLT2R2VFBEVESUQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.443234329 (558h15m3.213875664s) [noise=true , address=79.127.205.235:4666 , piece_id=DWLW664SQKO7VSUAG5MUEFQQ7T73FWZDFIERIPUAZUQ3XU5UMC4Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.721945067 (558h15m3.492586392s) [noise=true , address=45.91.95.3:4026 , piece_id=B3AKEM3ZYFOGMYP5XIAATWO64LMMANABZMYYI34HSIEMLXEYLZKA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.729714724 (558h15m3.500356059s) [noise=true , address=79.127.220.97:4370 , piece_id=FZXFXBB2JKYK4AB7RHOKGM336FDM6I5V4L7EWPMTBMF36VWPIUEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.748256620 (558h15m3.518897955s) [noise=true , address=45.91.95.5:4029 , piece_id=7E7CIOGMRUJBZ5TMUOXKESW233KAIXOQIYSCCY5EQV6CAPVGTVSA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.752952445 (558h15m3.52359378s) [noise=true , address=45.91.95.2:4030 , piece_id=GYVI274G7QRGELIWQJWSMJ6DYMS4L3INWCPARA7UALVB3AUTCF7Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.947992261 (558h15m3.718633597s) [noise=true , address=104.36.82.44:4888 , piece_id=BHRRID7B33YIVFBBP63O2PBX44EGSYEHEMLTVOKW75YWY5UQQTJQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:28.975520983 (558h15m3.746162318s) [noise=true , address=45.91.116.9:4287 , piece_id=HG2BREDTUCXHLD7ZQMTR3BQ5HJMH5XTI5NFK4DNHGFRFBV4MOQRA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.029579064 (558h15m3.800220399s) [noise=true , address=104.36.82.50:4027 , piece_id=XMFZ5XVXHJLX7UU4PAXSQEM7U7CZMNAXRQ7JMCI2SR2WTKZZR3LA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:25:29.136106708 (558h15m3.906748043s) [noise=true , address=79.127.219.38:4023 , piece_id=QTU5RUIRNKU7GOF54ORFIYR4VZVGBO5EBPGXVXDTP7BZ376ZDYEA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:00.871314102 (558h15m35.641955437s) [noise=true , address=77.90.31.17:4804 , piece_id=H45H2NQ3H7BT74QR2XBD5VKCF2MWVJOWBOWFC66FCFI7RR7UZ6SQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: <nil> Jul 16 15:26:25.907311351 (558h16m0.677952686s) [noise=true , address=77.90.31.21:4890 , piece_id=IGQJIH7PGGSQTP6WOS2X6NOE43RACFEAY623DYIQU3EHBNBOKAFQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1iE3mDnEhGDsMBtmUXC4EwvSrpyHLoybzVXkC9eoFuYEMHeBrd, address:77.90.31.21:4890): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907361918 (558h16m0.678003253s) [noise=true , address=77.90.31.13:4625 , piece_id=MKTO7FU6TIWGOE23USXCMHCZYY3WRNGBEJ3MKGPHXUV4VGARKIPQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1ZhwvtJXZsQbkcJpW92vWLC6qYHdcNzRk9gxV78ihgcGttTYVZ, address:77.90.31.13:4625): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907417575 (558h16m0.67805891s) [noise=true , address=77.90.31.11:4565 , piece_id=RVJS32QSKYKWS7ALW4C3WX35RM4YSR7DMUNZF2BUNJF6GDIVW6KA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1xnwFcUEdmpLjmx8NXZZcpiVQwTySXDRYbchuz4oREBq1yP6wh, address:77.90.31.11:4565): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907513367 (558h16m0.678154702s) [noise=true , address=77.90.31.19:4846 , piece_id=ZFAUXK5IKOGGYLVFHNYU7SSDZIO3CWIS3F5GU7UURSYUVV37H52Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 15:26:25.907603160 (558h16m0.678244495s) [noise=true , address=77.90.31.18:4824 , piece_id=U5TGOR35A74WKDMSH3VR6J4Q6YXDH7XSX5OB3IAZ4O3STNCWJU2Q, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53574->77.90.31.18:4824: use of closed network connection Jul 16 15:26:25.907644408 (558h16m0.678285743s) [noise=true , address=77.90.31.12:4595 , piece_id=3EWVKFFJXJ6KH7P26AB43JHBFGS6ENH2GNOFF5ZLAYXIJ556A4OA, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:1he15nyRz78fmHLkMRs8Hxi2HntmzNsPTqKEbHNUthhSjGMmuj, address:77.90.31.12:4595): protocol: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46410->77.90.31.12:4595: use of closed network connection Jul 16 15:26:25.907752425 (558h16m0.67839377s) [noise=true , address=77.90.31.10:4529 , piece_id=TNFHJRIP5A36CAT3ZJKRBZZJLELGJRGAQQJCXAGPI5TEVQ6PQRMQ, seg_pos={0 0}, upload=446427]: Done uploading piece. err: ecclient: upload failed (node:122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, address:77.90.31.10:4529): protocol: write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; write tcp 188.166.225.170:37468->77.90.31.10:4529: use of closed network connection; piecestore: piecestore close: context deadline exceeded Jul 16 15:26:25.907771041 (558h16m0.678412376s) [seg_pos={0 0}, upload=446427]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 16 15:26:26.144611036 (558h16m0.915252371s) [upload=446427]: Done uploading segments... Jul 16 15:26:26.144634341 (558h16m0.915275676s) [upload=446427]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.851820801s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 6.295221936s FAIL (2025-07-16 16:03:57 UTC) orbiter warning: context deadline exceeded Duration: 6.2952344s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........XXXX.............................................] duration: 5.689461856s FAIL (2025-07-16 16:03:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.689488216s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............XXXX............................................] duration: 5.686478285s FAIL (2025-07-16 16:03:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== 446459]: Uploading segments... Jul 16 15:27:54.011135252 (558h17m28.781776587s) [upload=446459]: Got next segment. Inline: false Jul 16 15:27:54.011190187 (558h17m28.781831522s) [upload=446459]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 15:27:54.011195288 (558h17m28.781836623s) [upload=446459]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 15:27:54.249183050 (558h17m29.019824385s) [seg_pos={0 0}, upload=446459]: Begin upload segment... Jul 16 15:27:54.249226833 (558h17m29.019868168s) [seg_pos={0 0}, upload=446459]: Done begin upload segment. Jul 16 15:27:54.249249406 (558h17m29.019890731s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249373975 (558h17m29.02001531s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249586871 (558h17m29.020228197s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249884030 (558h17m29.020525365s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.249904690 (558h17m29.020546025s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250152153 (558h17m29.020793489s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250299194 (558h17m29.02094053s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250416399 (558h17m29.021057735s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250584972 (558h17m29.021226297s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250628825 (558h17m29.02127016s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250831233 (558h17m29.021472568s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.250872171 (558h17m29.021513506s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251074769 (558h17m29.021716104s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251168909 (558h17m29.021810244s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251311421 (558h17m29.021952756s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:54.251511083 (558h17m29.022152418s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Uploading piece... Jul 16 15:27:55.955611246 (558h17m30.726252581s) [upload=446459]: Next returned nil segment Jul 16 15:27:55.955627608 (558h17m30.726268943s) [upload=446459]: Waiting for error group managing segments... Jul 16 15:27:56.296594443 (558h17m31.067235778s) [noise=true , address=104.36.82.2:4730 , piece_id=JWMLXOX2NJ6VJC7TIR2HKWXGEA7ZTUJWLZTZ7UC5KRQGDMRLQOFQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.302964752 (558h17m31.073606087s) [noise=true , address=45.91.116.6:4973 , piece_id=V4D53UKB7K3MLD52TFIL5PFERD3ADW6BPBRPMYEKHILOY3VPLW5Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.309510295 (558h17m31.08015165s) [noise=true , address=45.91.116.5:4298 , piece_id=EAISOTTBH5XTRZA2NCRVMZFJWQVH2JM62T24DOQPC6R72DUTYF6Q, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.312876716 (558h17m31.083518051s) [noise=true , address=45.91.116.7:4010 , piece_id=BNFIWBN7Z3VETDTY6PZYY7TSGPWW277JJ7WLPOHTBABLC273Z24A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.379700643 (558h17m31.150341978s) [noise=true , address=104.36.82.12:4149 , piece_id=V7LLT7FO6NTYXIGEUAD2RJOXOTDPLK3RQ626P7WXMMKUHW5LL35A, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383309578 (558h17m31.153950913s) [noise=true , address=45.91.116.12:4226 , piece_id=LRIIWX7DKRENYXZEU3USPLJSJZTRD6XNONG663XXAS3FWEFUS5BQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.383583172 (558h17m31.154224507s) [noise=true , address=45.91.116.9:4262 , piece_id=7WNRDSEHKDZOEOT2AGBZBTXOHRB4DCLSFCZFU7FV22O6FEBVOFJQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.397058584 (558h17m31.167699909s) [noise=true , address=104.36.82.13:4185 , piece_id=7NQE6AYB2WSPU5E2IMUU5FWZ6URYW3BJJRFRBL7BVBCPTZBBEUVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.400720041 (558h17m31.171361376s) [noise=true , address=45.91.116.11:4118 , piece_id=CVLDSUDJS6CUV3V5YH5KBRP57MS4LJSYPVELL6LJ5IA5RJOMI2VA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.401562562 (558h17m31.172203897s) [noise=true , address=45.91.116.10:4082 , piece_id=NFEDZ6HGLSE5KV47ONGJNEJKHYOX4CBDICHJBQLXRRDJAO2A3DTQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:27:56.408960107 (558h17m31.179601432s) [noise=true , address=45.91.116.11:4117 , piece_id=DJ3HQFJV5DVOHIQAWS5E4OZRRZMVCEJ3ML5JHSJO7XPZU4U7WJHQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: <nil> Jul 16 15:28:23.362578594 (558h17m58.133219929s) [noise=true , address=77.90.31.15:4697 , piece_id=NTSMJKYT3GMMCTNPTQGKLOYRR542K5COAHIZYFHPLXJZYTF66LKA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58114->77.90.31.15:4697: use of closed network connection Jul 16 15:28:23.362633359 (558h17m58.133274684s) [noise=true , address=77.90.31.12:4589 , piece_id=QQ675K4LSFVTA6NSNZ3Y6RRNJKRA7CIFHI2ZAYONMMUMQXBSI3HA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54556->77.90.31.12:4589: use of closed network connection Jul 16 15:28:23.362725565 (558h17m58.13336691s) [noise=true , address=77.90.31.13:4626 , piece_id=M3F4ZTOBYZ4CAV45SSJC3ZWM7YIRKBXOXROMZKRLMHGADQLONAVA, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51172->77.90.31.13:4626: use of closed network connection Jul 16 15:28:23.362752727 (558h17m58.133394062s) [noise=true , address=77.90.31.10:4519 , piece_id=IQBHXKB3YMG2YEBZYG5WTV3GBCTDNASKG2OO5IFLG6QT2NQHXKDQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, address:77.90.31.10:4519): protocol: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52012->77.90.31.10:4519: use of closed network connection Jul 16 15:28:23.362791751 (558h17m58.133433086s) [noise=true , address=77.90.31.18:4807 , piece_id=CF2DO2C6A3TUPHDR2OUCXJKTEG2KBCCPYAQQBXIDHBOAAO5GS5QQ, seg_pos={0 0}, upload=446459]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37046->77.90.31.18:4807: use of closed network connection Jul 16 15:28:23.362809095 (558h17m58.13345043s) [seg_pos={0 0}, upload=446459]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 16 15:28:23.599845446 (558h17m58.370486781s) [upload=446459]: Done uploading segments... Jul 16 15:28:23.599873319 (558h17m58.370514654s) [upload=446459]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.686498284s core uplink us-select-1

Orbiter Check History

checkers.HTTP{"GET" "https://www.storj-ipfs.com/ipfs/Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi", ExpectedContent: "Storj IPFS!"} failure (antiblip window 3/60): (ipfs.get)

checkers.HTTP{"GET" "https://www.storj-ipfs.com/ipfs/Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi", ExpectedContent: "Storj IPFS!"} failure (antiblip window 3/60): (ipfs.get)

checkers.HTTP{"GET" "https://www.storj-ipfs.com/ipfs/Qmbj74UUMUGDe4TES4fEbWEEe5fiwqJEjQ8CHaLJUF9dSi", ExpectedContent: "Storj IPFS!"} failure (antiblip window 3/60): (ipfs.get)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)