Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X..............X...................................] duration: 5.980466161s trace: https://jaeger.dev.storj.io/trace/1deda862aa4073e2 FAIL (2025-12-24 05:21:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:53:45.720403184 (445h24m45.591486219s) [upload=390995]: Starting upload Dec 24 04:53:45.720540295 (445h24m45.59162335s) [upload=390995]: Done starting upload Dec 24 04:53:45.720783568 (445h24m45.591866603s) [upload=390995]: Uploading segments... Dec 24 04:53:45.721560018 (445h24m45.592643083s) [upload=390995]: Got next segment. Inline: false Dec 24 04:53:45.721648986 (445h24m45.592732032s) [upload=390995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:53:45.721656201 (445h24m45.592739246s) [upload=390995]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:53:45.968344117 (445h24m45.839427152s) [seg_pos={0 0}, upload=390995]: Begin upload segment... Dec 24 04:53:45.968399892 (445h24m45.839482927s) [seg_pos={0 0}, upload=390995]: Done begin upload segment. Dec 24 04:53:45.968504732 (445h24m45.839587767s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.968571449 (445h24m45.839654484s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969513152 (445h24m45.840596187s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969756015 (445h24m45.84083905s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970245246 (445h24m45.841328281s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970597858 (445h24m45.841680903s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970970517 (445h24m45.842053553s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971362764 (445h24m45.842445799s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971712941 (445h24m45.842795986s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972101662 (445h24m45.843184697s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972560385 (445h24m45.84364342s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.974902435 (445h24m45.84598547s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.975744338 (445h24m45.846827374s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.976475501 (445h24m45.847558546s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.977774053 (445h24m45.848857098s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.978307899 (445h24m45.849390934s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:47.684950330 (445h24m47.556033365s) [upload=390995]: Next returned nil segment Dec 24 04:53:47.684967432 (445h24m47.556050477s) [upload=390995]: Waiting for error group managing segments... Dec 24 04:53:47.975268627 (445h24m47.846351662s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.976768513 (445h24m47.847851558s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.977666483 (445h24m47.848749528s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.979492480 (445h24m47.850575515s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.982702883 (445h24m47.853785919s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.990496404 (445h24m47.861579449s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.019123966 (445h24m47.890207001s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.205478756 (445h24m48.076561801s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.215568068 (445h24m48.086651114s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.255055000 (445h24m48.126138045s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.360364752 (445h24m48.231447787s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:14.990184584 (445h25m14.861267629s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:15.032082447 (445h25m14.903165492s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Dec 24 04:54:15.032375033 (445h25m14.903458078s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection Dec 24 04:54:15.032605533 (445h25m14.903688578s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, address:145.79.227.50:4375): protocol: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection Dec 24 04:54:15.032711815 (445h25m14.90379486s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection Dec 24 04:54:15.032736873 (445h25m14.903819908s) [seg_pos={0 0}, upload=390995]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:54:15.278462246 (445h25m15.149545291s) [upload=390995]: Done uploading segments... Dec 24 04:54:15.278513083 (445h25m15.149596138s) [upload=390995]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.980494365s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X..............X..................................] duration: 6.005750289s trace: https://jaeger.dev.storj.io/trace/1deda862aa4073e2 FAIL (2025-12-24 05:20:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:53:45.720403184 (445h24m45.591486219s) [upload=390995]: Starting upload Dec 24 04:53:45.720540295 (445h24m45.59162335s) [upload=390995]: Done starting upload Dec 24 04:53:45.720783568 (445h24m45.591866603s) [upload=390995]: Uploading segments... Dec 24 04:53:45.721560018 (445h24m45.592643083s) [upload=390995]: Got next segment. Inline: false Dec 24 04:53:45.721648986 (445h24m45.592732032s) [upload=390995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:53:45.721656201 (445h24m45.592739246s) [upload=390995]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:53:45.968344117 (445h24m45.839427152s) [seg_pos={0 0}, upload=390995]: Begin upload segment... Dec 24 04:53:45.968399892 (445h24m45.839482927s) [seg_pos={0 0}, upload=390995]: Done begin upload segment. Dec 24 04:53:45.968504732 (445h24m45.839587767s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.968571449 (445h24m45.839654484s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969513152 (445h24m45.840596187s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969756015 (445h24m45.84083905s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970245246 (445h24m45.841328281s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970597858 (445h24m45.841680903s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970970517 (445h24m45.842053553s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971362764 (445h24m45.842445799s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971712941 (445h24m45.842795986s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972101662 (445h24m45.843184697s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972560385 (445h24m45.84364342s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.974902435 (445h24m45.84598547s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.975744338 (445h24m45.846827374s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.976475501 (445h24m45.847558546s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.977774053 (445h24m45.848857098s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.978307899 (445h24m45.849390934s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:47.684950330 (445h24m47.556033365s) [upload=390995]: Next returned nil segment Dec 24 04:53:47.684967432 (445h24m47.556050477s) [upload=390995]: Waiting for error group managing segments... Dec 24 04:53:47.975268627 (445h24m47.846351662s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.976768513 (445h24m47.847851558s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.977666483 (445h24m47.848749528s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.979492480 (445h24m47.850575515s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.982702883 (445h24m47.853785919s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.990496404 (445h24m47.861579449s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.019123966 (445h24m47.890207001s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.205478756 (445h24m48.076561801s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.215568068 (445h24m48.086651114s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.255055000 (445h24m48.126138045s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.360364752 (445h24m48.231447787s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:14.990184584 (445h25m14.861267629s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:15.032082447 (445h25m14.903165492s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Dec 24 04:54:15.032375033 (445h25m14.903458078s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection Dec 24 04:54:15.032605533 (445h25m14.903688578s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, address:145.79.227.50:4375): protocol: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection Dec 24 04:54:15.032711815 (445h25m14.90379486s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection Dec 24 04:54:15.032736873 (445h25m14.903819908s) [seg_pos={0 0}, upload=390995]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:54:15.278462246 (445h25m15.149545291s) [upload=390995]: Done uploading segments... Dec 24 04:54:15.278513083 (445h25m15.149596138s) [upload=390995]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.005779735s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X..............X.................................] duration: 6.353181474s trace: https://jaeger.dev.storj.io/trace/1deda862aa4073e2 FAIL (2025-12-24 05:20:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:53:45.720403184 (445h24m45.591486219s) [upload=390995]: Starting upload Dec 24 04:53:45.720540295 (445h24m45.59162335s) [upload=390995]: Done starting upload Dec 24 04:53:45.720783568 (445h24m45.591866603s) [upload=390995]: Uploading segments... Dec 24 04:53:45.721560018 (445h24m45.592643083s) [upload=390995]: Got next segment. Inline: false Dec 24 04:53:45.721648986 (445h24m45.592732032s) [upload=390995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:53:45.721656201 (445h24m45.592739246s) [upload=390995]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:53:45.968344117 (445h24m45.839427152s) [seg_pos={0 0}, upload=390995]: Begin upload segment... Dec 24 04:53:45.968399892 (445h24m45.839482927s) [seg_pos={0 0}, upload=390995]: Done begin upload segment. Dec 24 04:53:45.968504732 (445h24m45.839587767s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.968571449 (445h24m45.839654484s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969513152 (445h24m45.840596187s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969756015 (445h24m45.84083905s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970245246 (445h24m45.841328281s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970597858 (445h24m45.841680903s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970970517 (445h24m45.842053553s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971362764 (445h24m45.842445799s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971712941 (445h24m45.842795986s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972101662 (445h24m45.843184697s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972560385 (445h24m45.84364342s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.974902435 (445h24m45.84598547s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.975744338 (445h24m45.846827374s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.976475501 (445h24m45.847558546s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.977774053 (445h24m45.848857098s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.978307899 (445h24m45.849390934s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:47.684950330 (445h24m47.556033365s) [upload=390995]: Next returned nil segment Dec 24 04:53:47.684967432 (445h24m47.556050477s) [upload=390995]: Waiting for error group managing segments... Dec 24 04:53:47.975268627 (445h24m47.846351662s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.976768513 (445h24m47.847851558s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.977666483 (445h24m47.848749528s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.979492480 (445h24m47.850575515s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.982702883 (445h24m47.853785919s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.990496404 (445h24m47.861579449s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.019123966 (445h24m47.890207001s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.205478756 (445h24m48.076561801s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.215568068 (445h24m48.086651114s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.255055000 (445h24m48.126138045s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.360364752 (445h24m48.231447787s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:14.990184584 (445h25m14.861267629s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:15.032082447 (445h25m14.903165492s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Dec 24 04:54:15.032375033 (445h25m14.903458078s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection Dec 24 04:54:15.032605533 (445h25m14.903688578s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, address:145.79.227.50:4375): protocol: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection Dec 24 04:54:15.032711815 (445h25m14.90379486s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection Dec 24 04:54:15.032736873 (445h25m14.903819908s) [seg_pos={0 0}, upload=390995]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:54:15.278462246 (445h25m15.149545291s) [upload=390995]: Done uploading segments... Dec 24 04:54:15.278513083 (445h25m15.149596138s) [upload=390995]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.353194359s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X..............X................................] duration: 6.112085362s trace: https://jaeger.dev.storj.io/trace/1deda862aa4073e2 FAIL (2025-12-24 05:19:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:53:45.720403184 (445h24m45.591486219s) [upload=390995]: Starting upload Dec 24 04:53:45.720540295 (445h24m45.59162335s) [upload=390995]: Done starting upload Dec 24 04:53:45.720783568 (445h24m45.591866603s) [upload=390995]: Uploading segments... Dec 24 04:53:45.721560018 (445h24m45.592643083s) [upload=390995]: Got next segment. Inline: false Dec 24 04:53:45.721648986 (445h24m45.592732032s) [upload=390995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:53:45.721656201 (445h24m45.592739246s) [upload=390995]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:53:45.968344117 (445h24m45.839427152s) [seg_pos={0 0}, upload=390995]: Begin upload segment... Dec 24 04:53:45.968399892 (445h24m45.839482927s) [seg_pos={0 0}, upload=390995]: Done begin upload segment. Dec 24 04:53:45.968504732 (445h24m45.839587767s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.968571449 (445h24m45.839654484s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969513152 (445h24m45.840596187s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969756015 (445h24m45.84083905s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970245246 (445h24m45.841328281s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970597858 (445h24m45.841680903s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970970517 (445h24m45.842053553s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971362764 (445h24m45.842445799s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971712941 (445h24m45.842795986s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972101662 (445h24m45.843184697s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972560385 (445h24m45.84364342s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.974902435 (445h24m45.84598547s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.975744338 (445h24m45.846827374s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.976475501 (445h24m45.847558546s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.977774053 (445h24m45.848857098s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.978307899 (445h24m45.849390934s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:47.684950330 (445h24m47.556033365s) [upload=390995]: Next returned nil segment Dec 24 04:53:47.684967432 (445h24m47.556050477s) [upload=390995]: Waiting for error group managing segments... Dec 24 04:53:47.975268627 (445h24m47.846351662s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.976768513 (445h24m47.847851558s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.977666483 (445h24m47.848749528s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.979492480 (445h24m47.850575515s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.982702883 (445h24m47.853785919s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.990496404 (445h24m47.861579449s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.019123966 (445h24m47.890207001s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.205478756 (445h24m48.076561801s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.215568068 (445h24m48.086651114s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.255055000 (445h24m48.126138045s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.360364752 (445h24m48.231447787s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:14.990184584 (445h25m14.861267629s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:15.032082447 (445h25m14.903165492s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Dec 24 04:54:15.032375033 (445h25m14.903458078s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection Dec 24 04:54:15.032605533 (445h25m14.903688578s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, address:145.79.227.50:4375): protocol: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection Dec 24 04:54:15.032711815 (445h25m14.90379486s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection Dec 24 04:54:15.032736873 (445h25m14.903819908s) [seg_pos={0 0}, upload=390995]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:54:15.278462246 (445h25m15.149545291s) [upload=390995]: Done uploading segments... Dec 24 04:54:15.278513083 (445h25m15.149596138s) [upload=390995]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.112113537s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X..............X...............................] duration: 5.858743886s trace: https://jaeger.dev.storj.io/trace/1deda862aa4073e2 FAIL (2025-12-24 05:18:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:53:45.720403184 (445h24m45.591486219s) [upload=390995]: Starting upload Dec 24 04:53:45.720540295 (445h24m45.59162335s) [upload=390995]: Done starting upload Dec 24 04:53:45.720783568 (445h24m45.591866603s) [upload=390995]: Uploading segments... Dec 24 04:53:45.721560018 (445h24m45.592643083s) [upload=390995]: Got next segment. Inline: false Dec 24 04:53:45.721648986 (445h24m45.592732032s) [upload=390995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:53:45.721656201 (445h24m45.592739246s) [upload=390995]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:53:45.968344117 (445h24m45.839427152s) [seg_pos={0 0}, upload=390995]: Begin upload segment... Dec 24 04:53:45.968399892 (445h24m45.839482927s) [seg_pos={0 0}, upload=390995]: Done begin upload segment. Dec 24 04:53:45.968504732 (445h24m45.839587767s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.968571449 (445h24m45.839654484s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969513152 (445h24m45.840596187s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969756015 (445h24m45.84083905s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970245246 (445h24m45.841328281s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970597858 (445h24m45.841680903s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970970517 (445h24m45.842053553s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971362764 (445h24m45.842445799s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971712941 (445h24m45.842795986s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972101662 (445h24m45.843184697s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972560385 (445h24m45.84364342s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.974902435 (445h24m45.84598547s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.975744338 (445h24m45.846827374s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.976475501 (445h24m45.847558546s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.977774053 (445h24m45.848857098s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.978307899 (445h24m45.849390934s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:47.684950330 (445h24m47.556033365s) [upload=390995]: Next returned nil segment Dec 24 04:53:47.684967432 (445h24m47.556050477s) [upload=390995]: Waiting for error group managing segments... Dec 24 04:53:47.975268627 (445h24m47.846351662s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.976768513 (445h24m47.847851558s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.977666483 (445h24m47.848749528s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.979492480 (445h24m47.850575515s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.982702883 (445h24m47.853785919s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.990496404 (445h24m47.861579449s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.019123966 (445h24m47.890207001s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.205478756 (445h24m48.076561801s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.215568068 (445h24m48.086651114s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.255055000 (445h24m48.126138045s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.360364752 (445h24m48.231447787s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:14.990184584 (445h25m14.861267629s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:15.032082447 (445h25m14.903165492s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Dec 24 04:54:15.032375033 (445h25m14.903458078s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection Dec 24 04:54:15.032605533 (445h25m14.903688578s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, address:145.79.227.50:4375): protocol: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection Dec 24 04:54:15.032711815 (445h25m14.90379486s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection Dec 24 04:54:15.032736873 (445h25m14.903819908s) [seg_pos={0 0}, upload=390995]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:54:15.278462246 (445h25m15.149545291s) [upload=390995]: Done uploading segments... Dec 24 04:54:15.278513083 (445h25m15.149596138s) [upload=390995]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.858772701s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X..............X..............................] duration: 10.338675004s trace: https://jaeger.dev.storj.io/trace/1deda862aa4073e2 FAIL (2025-12-24 05:18:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:53:45.720403184 (445h24m45.591486219s) [upload=390995]: Starting upload Dec 24 04:53:45.720540295 (445h24m45.59162335s) [upload=390995]: Done starting upload Dec 24 04:53:45.720783568 (445h24m45.591866603s) [upload=390995]: Uploading segments... Dec 24 04:53:45.721560018 (445h24m45.592643083s) [upload=390995]: Got next segment. Inline: false Dec 24 04:53:45.721648986 (445h24m45.592732032s) [upload=390995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:53:45.721656201 (445h24m45.592739246s) [upload=390995]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:53:45.968344117 (445h24m45.839427152s) [seg_pos={0 0}, upload=390995]: Begin upload segment... Dec 24 04:53:45.968399892 (445h24m45.839482927s) [seg_pos={0 0}, upload=390995]: Done begin upload segment. Dec 24 04:53:45.968504732 (445h24m45.839587767s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.968571449 (445h24m45.839654484s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969513152 (445h24m45.840596187s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969756015 (445h24m45.84083905s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970245246 (445h24m45.841328281s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970597858 (445h24m45.841680903s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970970517 (445h24m45.842053553s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971362764 (445h24m45.842445799s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971712941 (445h24m45.842795986s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972101662 (445h24m45.843184697s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972560385 (445h24m45.84364342s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.974902435 (445h24m45.84598547s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.975744338 (445h24m45.846827374s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.976475501 (445h24m45.847558546s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.977774053 (445h24m45.848857098s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.978307899 (445h24m45.849390934s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:47.684950330 (445h24m47.556033365s) [upload=390995]: Next returned nil segment Dec 24 04:53:47.684967432 (445h24m47.556050477s) [upload=390995]: Waiting for error group managing segments... Dec 24 04:53:47.975268627 (445h24m47.846351662s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.976768513 (445h24m47.847851558s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.977666483 (445h24m47.848749528s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.979492480 (445h24m47.850575515s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.982702883 (445h24m47.853785919s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.990496404 (445h24m47.861579449s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.019123966 (445h24m47.890207001s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.205478756 (445h24m48.076561801s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.215568068 (445h24m48.086651114s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.255055000 (445h24m48.126138045s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.360364752 (445h24m48.231447787s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:14.990184584 (445h25m14.861267629s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:15.032082447 (445h25m14.903165492s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Dec 24 04:54:15.032375033 (445h25m14.903458078s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection Dec 24 04:54:15.032605533 (445h25m14.903688578s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, address:145.79.227.50:4375): protocol: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection Dec 24 04:54:15.032711815 (445h25m14.90379486s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection Dec 24 04:54:15.032736873 (445h25m14.903819908s) [seg_pos={0 0}, upload=390995]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:54:15.278462246 (445h25m15.149545291s) [upload=390995]: Done uploading segments... Dec 24 04:54:15.278513083 (445h25m15.149596138s) [upload=390995]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 10.338703577s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X..............X.............................] duration: 6.204758313s trace: https://jaeger.dev.storj.io/trace/1deda862aa4073e2 FAIL (2025-12-24 05:17:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:53:45.720403184 (445h24m45.591486219s) [upload=390995]: Starting upload Dec 24 04:53:45.720540295 (445h24m45.59162335s) [upload=390995]: Done starting upload Dec 24 04:53:45.720783568 (445h24m45.591866603s) [upload=390995]: Uploading segments... Dec 24 04:53:45.721560018 (445h24m45.592643083s) [upload=390995]: Got next segment. Inline: false Dec 24 04:53:45.721648986 (445h24m45.592732032s) [upload=390995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:53:45.721656201 (445h24m45.592739246s) [upload=390995]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:53:45.968344117 (445h24m45.839427152s) [seg_pos={0 0}, upload=390995]: Begin upload segment... Dec 24 04:53:45.968399892 (445h24m45.839482927s) [seg_pos={0 0}, upload=390995]: Done begin upload segment. Dec 24 04:53:45.968504732 (445h24m45.839587767s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.968571449 (445h24m45.839654484s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969513152 (445h24m45.840596187s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969756015 (445h24m45.84083905s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970245246 (445h24m45.841328281s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970597858 (445h24m45.841680903s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970970517 (445h24m45.842053553s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971362764 (445h24m45.842445799s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971712941 (445h24m45.842795986s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972101662 (445h24m45.843184697s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972560385 (445h24m45.84364342s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.974902435 (445h24m45.84598547s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.975744338 (445h24m45.846827374s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.976475501 (445h24m45.847558546s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.977774053 (445h24m45.848857098s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.978307899 (445h24m45.849390934s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:47.684950330 (445h24m47.556033365s) [upload=390995]: Next returned nil segment Dec 24 04:53:47.684967432 (445h24m47.556050477s) [upload=390995]: Waiting for error group managing segments... Dec 24 04:53:47.975268627 (445h24m47.846351662s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.976768513 (445h24m47.847851558s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.977666483 (445h24m47.848749528s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.979492480 (445h24m47.850575515s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.982702883 (445h24m47.853785919s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.990496404 (445h24m47.861579449s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.019123966 (445h24m47.890207001s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.205478756 (445h24m48.076561801s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.215568068 (445h24m48.086651114s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.255055000 (445h24m48.126138045s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.360364752 (445h24m48.231447787s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:14.990184584 (445h25m14.861267629s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:15.032082447 (445h25m14.903165492s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Dec 24 04:54:15.032375033 (445h25m14.903458078s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection Dec 24 04:54:15.032605533 (445h25m14.903688578s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, address:145.79.227.50:4375): protocol: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection Dec 24 04:54:15.032711815 (445h25m14.90379486s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection Dec 24 04:54:15.032736873 (445h25m14.903819908s) [seg_pos={0 0}, upload=390995]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:54:15.278462246 (445h25m15.149545291s) [upload=390995]: Done uploading segments... Dec 24 04:54:15.278513083 (445h25m15.149596138s) [upload=390995]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.204785453s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X..............X............................] duration: 6.334764824s trace: https://jaeger.dev.storj.io/trace/1deda862aa4073e2 FAIL (2025-12-24 05:16:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:53:45.720403184 (445h24m45.591486219s) [upload=390995]: Starting upload Dec 24 04:53:45.720540295 (445h24m45.59162335s) [upload=390995]: Done starting upload Dec 24 04:53:45.720783568 (445h24m45.591866603s) [upload=390995]: Uploading segments... Dec 24 04:53:45.721560018 (445h24m45.592643083s) [upload=390995]: Got next segment. Inline: false Dec 24 04:53:45.721648986 (445h24m45.592732032s) [upload=390995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:53:45.721656201 (445h24m45.592739246s) [upload=390995]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:53:45.968344117 (445h24m45.839427152s) [seg_pos={0 0}, upload=390995]: Begin upload segment... Dec 24 04:53:45.968399892 (445h24m45.839482927s) [seg_pos={0 0}, upload=390995]: Done begin upload segment. Dec 24 04:53:45.968504732 (445h24m45.839587767s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.968571449 (445h24m45.839654484s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969513152 (445h24m45.840596187s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969756015 (445h24m45.84083905s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970245246 (445h24m45.841328281s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970597858 (445h24m45.841680903s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970970517 (445h24m45.842053553s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971362764 (445h24m45.842445799s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971712941 (445h24m45.842795986s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972101662 (445h24m45.843184697s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972560385 (445h24m45.84364342s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.974902435 (445h24m45.84598547s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.975744338 (445h24m45.846827374s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.976475501 (445h24m45.847558546s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.977774053 (445h24m45.848857098s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.978307899 (445h24m45.849390934s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:47.684950330 (445h24m47.556033365s) [upload=390995]: Next returned nil segment Dec 24 04:53:47.684967432 (445h24m47.556050477s) [upload=390995]: Waiting for error group managing segments... Dec 24 04:53:47.975268627 (445h24m47.846351662s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.976768513 (445h24m47.847851558s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.977666483 (445h24m47.848749528s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.979492480 (445h24m47.850575515s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.982702883 (445h24m47.853785919s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.990496404 (445h24m47.861579449s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.019123966 (445h24m47.890207001s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.205478756 (445h24m48.076561801s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.215568068 (445h24m48.086651114s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.255055000 (445h24m48.126138045s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.360364752 (445h24m48.231447787s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:14.990184584 (445h25m14.861267629s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:15.032082447 (445h25m14.903165492s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Dec 24 04:54:15.032375033 (445h25m14.903458078s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection Dec 24 04:54:15.032605533 (445h25m14.903688578s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, address:145.79.227.50:4375): protocol: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection Dec 24 04:54:15.032711815 (445h25m14.90379486s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection Dec 24 04:54:15.032736873 (445h25m14.903819908s) [seg_pos={0 0}, upload=390995]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:54:15.278462246 (445h25m15.149545291s) [upload=390995]: Done uploading segments... Dec 24 04:54:15.278513083 (445h25m15.149596138s) [upload=390995]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.334801865s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X..............X...........................] duration: 5.828347697s trace: https://jaeger.dev.storj.io/trace/1deda862aa4073e2 FAIL (2025-12-24 05:15:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:53:45.720403184 (445h24m45.591486219s) [upload=390995]: Starting upload Dec 24 04:53:45.720540295 (445h24m45.59162335s) [upload=390995]: Done starting upload Dec 24 04:53:45.720783568 (445h24m45.591866603s) [upload=390995]: Uploading segments... Dec 24 04:53:45.721560018 (445h24m45.592643083s) [upload=390995]: Got next segment. Inline: false Dec 24 04:53:45.721648986 (445h24m45.592732032s) [upload=390995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:53:45.721656201 (445h24m45.592739246s) [upload=390995]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:53:45.968344117 (445h24m45.839427152s) [seg_pos={0 0}, upload=390995]: Begin upload segment... Dec 24 04:53:45.968399892 (445h24m45.839482927s) [seg_pos={0 0}, upload=390995]: Done begin upload segment. Dec 24 04:53:45.968504732 (445h24m45.839587767s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.968571449 (445h24m45.839654484s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969513152 (445h24m45.840596187s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969756015 (445h24m45.84083905s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970245246 (445h24m45.841328281s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970597858 (445h24m45.841680903s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970970517 (445h24m45.842053553s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971362764 (445h24m45.842445799s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971712941 (445h24m45.842795986s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972101662 (445h24m45.843184697s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972560385 (445h24m45.84364342s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.974902435 (445h24m45.84598547s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.975744338 (445h24m45.846827374s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.976475501 (445h24m45.847558546s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.977774053 (445h24m45.848857098s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.978307899 (445h24m45.849390934s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:47.684950330 (445h24m47.556033365s) [upload=390995]: Next returned nil segment Dec 24 04:53:47.684967432 (445h24m47.556050477s) [upload=390995]: Waiting for error group managing segments... Dec 24 04:53:47.975268627 (445h24m47.846351662s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.976768513 (445h24m47.847851558s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.977666483 (445h24m47.848749528s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.979492480 (445h24m47.850575515s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.982702883 (445h24m47.853785919s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.990496404 (445h24m47.861579449s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.019123966 (445h24m47.890207001s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.205478756 (445h24m48.076561801s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.215568068 (445h24m48.086651114s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.255055000 (445h24m48.126138045s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.360364752 (445h24m48.231447787s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:14.990184584 (445h25m14.861267629s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:15.032082447 (445h25m14.903165492s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Dec 24 04:54:15.032375033 (445h25m14.903458078s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection Dec 24 04:54:15.032605533 (445h25m14.903688578s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, address:145.79.227.50:4375): protocol: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection Dec 24 04:54:15.032711815 (445h25m14.90379486s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection Dec 24 04:54:15.032736873 (445h25m14.903819908s) [seg_pos={0 0}, upload=390995]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:54:15.278462246 (445h25m15.149545291s) [upload=390995]: Done uploading segments... Dec 24 04:54:15.278513083 (445h25m15.149596138s) [upload=390995]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.828378295s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X..............X..........................] duration: 5.674182501s trace: https://jaeger.dev.storj.io/trace/1deda862aa4073e2 FAIL (2025-12-24 05:15:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:53:45.720403184 (445h24m45.591486219s) [upload=390995]: Starting upload Dec 24 04:53:45.720540295 (445h24m45.59162335s) [upload=390995]: Done starting upload Dec 24 04:53:45.720783568 (445h24m45.591866603s) [upload=390995]: Uploading segments... Dec 24 04:53:45.721560018 (445h24m45.592643083s) [upload=390995]: Got next segment. Inline: false Dec 24 04:53:45.721648986 (445h24m45.592732032s) [upload=390995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:53:45.721656201 (445h24m45.592739246s) [upload=390995]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:53:45.968344117 (445h24m45.839427152s) [seg_pos={0 0}, upload=390995]: Begin upload segment... Dec 24 04:53:45.968399892 (445h24m45.839482927s) [seg_pos={0 0}, upload=390995]: Done begin upload segment. Dec 24 04:53:45.968504732 (445h24m45.839587767s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.968571449 (445h24m45.839654484s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969513152 (445h24m45.840596187s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969756015 (445h24m45.84083905s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970245246 (445h24m45.841328281s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970597858 (445h24m45.841680903s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970970517 (445h24m45.842053553s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971362764 (445h24m45.842445799s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971712941 (445h24m45.842795986s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972101662 (445h24m45.843184697s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972560385 (445h24m45.84364342s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.974902435 (445h24m45.84598547s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.975744338 (445h24m45.846827374s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.976475501 (445h24m45.847558546s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.977774053 (445h24m45.848857098s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.978307899 (445h24m45.849390934s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:47.684950330 (445h24m47.556033365s) [upload=390995]: Next returned nil segment Dec 24 04:53:47.684967432 (445h24m47.556050477s) [upload=390995]: Waiting for error group managing segments... Dec 24 04:53:47.975268627 (445h24m47.846351662s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.976768513 (445h24m47.847851558s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.977666483 (445h24m47.848749528s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.979492480 (445h24m47.850575515s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.982702883 (445h24m47.853785919s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.990496404 (445h24m47.861579449s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.019123966 (445h24m47.890207001s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.205478756 (445h24m48.076561801s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.215568068 (445h24m48.086651114s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.255055000 (445h24m48.126138045s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.360364752 (445h24m48.231447787s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:14.990184584 (445h25m14.861267629s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:15.032082447 (445h25m14.903165492s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Dec 24 04:54:15.032375033 (445h25m14.903458078s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection Dec 24 04:54:15.032605533 (445h25m14.903688578s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, address:145.79.227.50:4375): protocol: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection Dec 24 04:54:15.032711815 (445h25m14.90379486s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection Dec 24 04:54:15.032736873 (445h25m14.903819908s) [seg_pos={0 0}, upload=390995]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:54:15.278462246 (445h25m15.149545291s) [upload=390995]: Done uploading segments... Dec 24 04:54:15.278513083 (445h25m15.149596138s) [upload=390995]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.674224151s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X..............X.........................] duration: 5.656009876s trace: https://jaeger.dev.storj.io/trace/1deda862aa4073e2 FAIL (2025-12-24 05:14:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:53:45.720403184 (445h24m45.591486219s) [upload=390995]: Starting upload Dec 24 04:53:45.720540295 (445h24m45.59162335s) [upload=390995]: Done starting upload Dec 24 04:53:45.720783568 (445h24m45.591866603s) [upload=390995]: Uploading segments... Dec 24 04:53:45.721560018 (445h24m45.592643083s) [upload=390995]: Got next segment. Inline: false Dec 24 04:53:45.721648986 (445h24m45.592732032s) [upload=390995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:53:45.721656201 (445h24m45.592739246s) [upload=390995]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:53:45.968344117 (445h24m45.839427152s) [seg_pos={0 0}, upload=390995]: Begin upload segment... Dec 24 04:53:45.968399892 (445h24m45.839482927s) [seg_pos={0 0}, upload=390995]: Done begin upload segment. Dec 24 04:53:45.968504732 (445h24m45.839587767s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.968571449 (445h24m45.839654484s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969513152 (445h24m45.840596187s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969756015 (445h24m45.84083905s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970245246 (445h24m45.841328281s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970597858 (445h24m45.841680903s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970970517 (445h24m45.842053553s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971362764 (445h24m45.842445799s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971712941 (445h24m45.842795986s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972101662 (445h24m45.843184697s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972560385 (445h24m45.84364342s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.974902435 (445h24m45.84598547s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.975744338 (445h24m45.846827374s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.976475501 (445h24m45.847558546s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.977774053 (445h24m45.848857098s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.978307899 (445h24m45.849390934s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:47.684950330 (445h24m47.556033365s) [upload=390995]: Next returned nil segment Dec 24 04:53:47.684967432 (445h24m47.556050477s) [upload=390995]: Waiting for error group managing segments... Dec 24 04:53:47.975268627 (445h24m47.846351662s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.976768513 (445h24m47.847851558s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.977666483 (445h24m47.848749528s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.979492480 (445h24m47.850575515s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.982702883 (445h24m47.853785919s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.990496404 (445h24m47.861579449s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.019123966 (445h24m47.890207001s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.205478756 (445h24m48.076561801s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.215568068 (445h24m48.086651114s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.255055000 (445h24m48.126138045s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.360364752 (445h24m48.231447787s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:14.990184584 (445h25m14.861267629s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:15.032082447 (445h25m14.903165492s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Dec 24 04:54:15.032375033 (445h25m14.903458078s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection Dec 24 04:54:15.032605533 (445h25m14.903688578s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, address:145.79.227.50:4375): protocol: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection Dec 24 04:54:15.032711815 (445h25m14.90379486s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection Dec 24 04:54:15.032736873 (445h25m14.903819908s) [seg_pos={0 0}, upload=390995]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:54:15.278462246 (445h25m15.149545291s) [upload=390995]: Done uploading segments... Dec 24 04:54:15.278513083 (445h25m15.149596138s) [upload=390995]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.656041017s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X..............X........................] duration: 5.692029825s trace: https://jaeger.dev.storj.io/trace/1deda862aa4073e2 FAIL (2025-12-24 05:13:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:53:45.720403184 (445h24m45.591486219s) [upload=390995]: Starting upload Dec 24 04:53:45.720540295 (445h24m45.59162335s) [upload=390995]: Done starting upload Dec 24 04:53:45.720783568 (445h24m45.591866603s) [upload=390995]: Uploading segments... Dec 24 04:53:45.721560018 (445h24m45.592643083s) [upload=390995]: Got next segment. Inline: false Dec 24 04:53:45.721648986 (445h24m45.592732032s) [upload=390995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:53:45.721656201 (445h24m45.592739246s) [upload=390995]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:53:45.968344117 (445h24m45.839427152s) [seg_pos={0 0}, upload=390995]: Begin upload segment... Dec 24 04:53:45.968399892 (445h24m45.839482927s) [seg_pos={0 0}, upload=390995]: Done begin upload segment. Dec 24 04:53:45.968504732 (445h24m45.839587767s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.968571449 (445h24m45.839654484s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969513152 (445h24m45.840596187s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969756015 (445h24m45.84083905s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970245246 (445h24m45.841328281s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970597858 (445h24m45.841680903s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970970517 (445h24m45.842053553s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971362764 (445h24m45.842445799s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971712941 (445h24m45.842795986s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972101662 (445h24m45.843184697s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972560385 (445h24m45.84364342s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.974902435 (445h24m45.84598547s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.975744338 (445h24m45.846827374s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.976475501 (445h24m45.847558546s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.977774053 (445h24m45.848857098s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.978307899 (445h24m45.849390934s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:47.684950330 (445h24m47.556033365s) [upload=390995]: Next returned nil segment Dec 24 04:53:47.684967432 (445h24m47.556050477s) [upload=390995]: Waiting for error group managing segments... Dec 24 04:53:47.975268627 (445h24m47.846351662s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.976768513 (445h24m47.847851558s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.977666483 (445h24m47.848749528s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.979492480 (445h24m47.850575515s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.982702883 (445h24m47.853785919s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.990496404 (445h24m47.861579449s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.019123966 (445h24m47.890207001s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.205478756 (445h24m48.076561801s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.215568068 (445h24m48.086651114s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.255055000 (445h24m48.126138045s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.360364752 (445h24m48.231447787s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:14.990184584 (445h25m14.861267629s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:15.032082447 (445h25m14.903165492s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Dec 24 04:54:15.032375033 (445h25m14.903458078s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection Dec 24 04:54:15.032605533 (445h25m14.903688578s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, address:145.79.227.50:4375): protocol: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection Dec 24 04:54:15.032711815 (445h25m14.90379486s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection Dec 24 04:54:15.032736873 (445h25m14.903819908s) [seg_pos={0 0}, upload=390995]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:54:15.278462246 (445h25m15.149545291s) [upload=390995]: Done uploading segments... Dec 24 04:54:15.278513083 (445h25m15.149596138s) [upload=390995]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.69205784s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X..............X.......................] duration: 6.124290686s trace: https://jaeger.dev.storj.io/trace/1deda862aa4073e2 FAIL (2025-12-24 05:13:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:53:45.720403184 (445h24m45.591486219s) [upload=390995]: Starting upload Dec 24 04:53:45.720540295 (445h24m45.59162335s) [upload=390995]: Done starting upload Dec 24 04:53:45.720783568 (445h24m45.591866603s) [upload=390995]: Uploading segments... Dec 24 04:53:45.721560018 (445h24m45.592643083s) [upload=390995]: Got next segment. Inline: false Dec 24 04:53:45.721648986 (445h24m45.592732032s) [upload=390995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:53:45.721656201 (445h24m45.592739246s) [upload=390995]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:53:45.968344117 (445h24m45.839427152s) [seg_pos={0 0}, upload=390995]: Begin upload segment... Dec 24 04:53:45.968399892 (445h24m45.839482927s) [seg_pos={0 0}, upload=390995]: Done begin upload segment. Dec 24 04:53:45.968504732 (445h24m45.839587767s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.968571449 (445h24m45.839654484s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969513152 (445h24m45.840596187s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969756015 (445h24m45.84083905s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970245246 (445h24m45.841328281s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970597858 (445h24m45.841680903s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970970517 (445h24m45.842053553s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971362764 (445h24m45.842445799s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971712941 (445h24m45.842795986s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972101662 (445h24m45.843184697s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972560385 (445h24m45.84364342s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.974902435 (445h24m45.84598547s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.975744338 (445h24m45.846827374s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.976475501 (445h24m45.847558546s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.977774053 (445h24m45.848857098s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.978307899 (445h24m45.849390934s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:47.684950330 (445h24m47.556033365s) [upload=390995]: Next returned nil segment Dec 24 04:53:47.684967432 (445h24m47.556050477s) [upload=390995]: Waiting for error group managing segments... Dec 24 04:53:47.975268627 (445h24m47.846351662s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.976768513 (445h24m47.847851558s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.977666483 (445h24m47.848749528s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.979492480 (445h24m47.850575515s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.982702883 (445h24m47.853785919s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.990496404 (445h24m47.861579449s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.019123966 (445h24m47.890207001s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.205478756 (445h24m48.076561801s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.215568068 (445h24m48.086651114s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.255055000 (445h24m48.126138045s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.360364752 (445h24m48.231447787s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:14.990184584 (445h25m14.861267629s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:15.032082447 (445h25m14.903165492s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Dec 24 04:54:15.032375033 (445h25m14.903458078s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection Dec 24 04:54:15.032605533 (445h25m14.903688578s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, address:145.79.227.50:4375): protocol: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection Dec 24 04:54:15.032711815 (445h25m14.90379486s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection Dec 24 04:54:15.032736873 (445h25m14.903819908s) [seg_pos={0 0}, upload=390995]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:54:15.278462246 (445h25m15.149545291s) [upload=390995]: Done uploading segments... Dec 24 04:54:15.278513083 (445h25m15.149596138s) [upload=390995]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.124327226s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X..............X......................] duration: 5.734287193s trace: https://jaeger.dev.storj.io/trace/1deda862aa4073e2 FAIL (2025-12-24 05:12:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:53:45.720403184 (445h24m45.591486219s) [upload=390995]: Starting upload Dec 24 04:53:45.720540295 (445h24m45.59162335s) [upload=390995]: Done starting upload Dec 24 04:53:45.720783568 (445h24m45.591866603s) [upload=390995]: Uploading segments... Dec 24 04:53:45.721560018 (445h24m45.592643083s) [upload=390995]: Got next segment. Inline: false Dec 24 04:53:45.721648986 (445h24m45.592732032s) [upload=390995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:53:45.721656201 (445h24m45.592739246s) [upload=390995]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:53:45.968344117 (445h24m45.839427152s) [seg_pos={0 0}, upload=390995]: Begin upload segment... Dec 24 04:53:45.968399892 (445h24m45.839482927s) [seg_pos={0 0}, upload=390995]: Done begin upload segment. Dec 24 04:53:45.968504732 (445h24m45.839587767s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.968571449 (445h24m45.839654484s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969513152 (445h24m45.840596187s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969756015 (445h24m45.84083905s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970245246 (445h24m45.841328281s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970597858 (445h24m45.841680903s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970970517 (445h24m45.842053553s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971362764 (445h24m45.842445799s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971712941 (445h24m45.842795986s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972101662 (445h24m45.843184697s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972560385 (445h24m45.84364342s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.974902435 (445h24m45.84598547s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.975744338 (445h24m45.846827374s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.976475501 (445h24m45.847558546s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.977774053 (445h24m45.848857098s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.978307899 (445h24m45.849390934s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:47.684950330 (445h24m47.556033365s) [upload=390995]: Next returned nil segment Dec 24 04:53:47.684967432 (445h24m47.556050477s) [upload=390995]: Waiting for error group managing segments... Dec 24 04:53:47.975268627 (445h24m47.846351662s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.976768513 (445h24m47.847851558s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.977666483 (445h24m47.848749528s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.979492480 (445h24m47.850575515s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.982702883 (445h24m47.853785919s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.990496404 (445h24m47.861579449s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.019123966 (445h24m47.890207001s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.205478756 (445h24m48.076561801s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.215568068 (445h24m48.086651114s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.255055000 (445h24m48.126138045s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.360364752 (445h24m48.231447787s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:14.990184584 (445h25m14.861267629s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:15.032082447 (445h25m14.903165492s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Dec 24 04:54:15.032375033 (445h25m14.903458078s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection Dec 24 04:54:15.032605533 (445h25m14.903688578s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, address:145.79.227.50:4375): protocol: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection Dec 24 04:54:15.032711815 (445h25m14.90379486s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection Dec 24 04:54:15.032736873 (445h25m14.903819908s) [seg_pos={0 0}, upload=390995]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:54:15.278462246 (445h25m15.149545291s) [upload=390995]: Done uploading segments... Dec 24 04:54:15.278513083 (445h25m15.149596138s) [upload=390995]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.734306511s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X..............X.....................] duration: 6.064925844s trace: https://jaeger.dev.storj.io/trace/1deda862aa4073e2 FAIL (2025-12-24 05:11:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:53:45.720403184 (445h24m45.591486219s) [upload=390995]: Starting upload Dec 24 04:53:45.720540295 (445h24m45.59162335s) [upload=390995]: Done starting upload Dec 24 04:53:45.720783568 (445h24m45.591866603s) [upload=390995]: Uploading segments... Dec 24 04:53:45.721560018 (445h24m45.592643083s) [upload=390995]: Got next segment. Inline: false Dec 24 04:53:45.721648986 (445h24m45.592732032s) [upload=390995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:53:45.721656201 (445h24m45.592739246s) [upload=390995]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:53:45.968344117 (445h24m45.839427152s) [seg_pos={0 0}, upload=390995]: Begin upload segment... Dec 24 04:53:45.968399892 (445h24m45.839482927s) [seg_pos={0 0}, upload=390995]: Done begin upload segment. Dec 24 04:53:45.968504732 (445h24m45.839587767s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.968571449 (445h24m45.839654484s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969513152 (445h24m45.840596187s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969756015 (445h24m45.84083905s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970245246 (445h24m45.841328281s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970597858 (445h24m45.841680903s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970970517 (445h24m45.842053553s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971362764 (445h24m45.842445799s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971712941 (445h24m45.842795986s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972101662 (445h24m45.843184697s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972560385 (445h24m45.84364342s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.974902435 (445h24m45.84598547s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.975744338 (445h24m45.846827374s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.976475501 (445h24m45.847558546s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.977774053 (445h24m45.848857098s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.978307899 (445h24m45.849390934s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:47.684950330 (445h24m47.556033365s) [upload=390995]: Next returned nil segment Dec 24 04:53:47.684967432 (445h24m47.556050477s) [upload=390995]: Waiting for error group managing segments... Dec 24 04:53:47.975268627 (445h24m47.846351662s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.976768513 (445h24m47.847851558s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.977666483 (445h24m47.848749528s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.979492480 (445h24m47.850575515s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.982702883 (445h24m47.853785919s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.990496404 (445h24m47.861579449s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.019123966 (445h24m47.890207001s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.205478756 (445h24m48.076561801s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.215568068 (445h24m48.086651114s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.255055000 (445h24m48.126138045s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.360364752 (445h24m48.231447787s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:14.990184584 (445h25m14.861267629s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:15.032082447 (445h25m14.903165492s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Dec 24 04:54:15.032375033 (445h25m14.903458078s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection Dec 24 04:54:15.032605533 (445h25m14.903688578s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, address:145.79.227.50:4375): protocol: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection Dec 24 04:54:15.032711815 (445h25m14.90379486s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection Dec 24 04:54:15.032736873 (445h25m14.903819908s) [seg_pos={0 0}, upload=390995]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:54:15.278462246 (445h25m15.149545291s) [upload=390995]: Done uploading segments... Dec 24 04:54:15.278513083 (445h25m15.149596138s) [upload=390995]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.06494992s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X..............X....................] duration: 5.745775639s trace: https://jaeger.dev.storj.io/trace/1deda862aa4073e2 FAIL (2025-12-24 05:10:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:53:45.720403184 (445h24m45.591486219s) [upload=390995]: Starting upload Dec 24 04:53:45.720540295 (445h24m45.59162335s) [upload=390995]: Done starting upload Dec 24 04:53:45.720783568 (445h24m45.591866603s) [upload=390995]: Uploading segments... Dec 24 04:53:45.721560018 (445h24m45.592643083s) [upload=390995]: Got next segment. Inline: false Dec 24 04:53:45.721648986 (445h24m45.592732032s) [upload=390995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:53:45.721656201 (445h24m45.592739246s) [upload=390995]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:53:45.968344117 (445h24m45.839427152s) [seg_pos={0 0}, upload=390995]: Begin upload segment... Dec 24 04:53:45.968399892 (445h24m45.839482927s) [seg_pos={0 0}, upload=390995]: Done begin upload segment. Dec 24 04:53:45.968504732 (445h24m45.839587767s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.968571449 (445h24m45.839654484s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969513152 (445h24m45.840596187s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969756015 (445h24m45.84083905s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970245246 (445h24m45.841328281s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970597858 (445h24m45.841680903s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970970517 (445h24m45.842053553s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971362764 (445h24m45.842445799s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971712941 (445h24m45.842795986s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972101662 (445h24m45.843184697s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972560385 (445h24m45.84364342s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.974902435 (445h24m45.84598547s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.975744338 (445h24m45.846827374s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.976475501 (445h24m45.847558546s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.977774053 (445h24m45.848857098s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.978307899 (445h24m45.849390934s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:47.684950330 (445h24m47.556033365s) [upload=390995]: Next returned nil segment Dec 24 04:53:47.684967432 (445h24m47.556050477s) [upload=390995]: Waiting for error group managing segments... Dec 24 04:53:47.975268627 (445h24m47.846351662s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.976768513 (445h24m47.847851558s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.977666483 (445h24m47.848749528s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.979492480 (445h24m47.850575515s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.982702883 (445h24m47.853785919s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.990496404 (445h24m47.861579449s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.019123966 (445h24m47.890207001s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.205478756 (445h24m48.076561801s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.215568068 (445h24m48.086651114s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.255055000 (445h24m48.126138045s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.360364752 (445h24m48.231447787s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:14.990184584 (445h25m14.861267629s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:15.032082447 (445h25m14.903165492s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Dec 24 04:54:15.032375033 (445h25m14.903458078s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection Dec 24 04:54:15.032605533 (445h25m14.903688578s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, address:145.79.227.50:4375): protocol: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection Dec 24 04:54:15.032711815 (445h25m14.90379486s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection Dec 24 04:54:15.032736873 (445h25m14.903819908s) [seg_pos={0 0}, upload=390995]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:54:15.278462246 (445h25m15.149545291s) [upload=390995]: Done uploading segments... Dec 24 04:54:15.278513083 (445h25m15.149596138s) [upload=390995]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.74595013s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X..............X...................] duration: 20.277344096s trace: https://jaeger.dev.storj.io/trace/1deda862aa4073e2 FAIL (2025-12-24 05:09:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:53:45.720403184 (445h24m45.591486219s) [upload=390995]: Starting upload Dec 24 04:53:45.720540295 (445h24m45.59162335s) [upload=390995]: Done starting upload Dec 24 04:53:45.720783568 (445h24m45.591866603s) [upload=390995]: Uploading segments... Dec 24 04:53:45.721560018 (445h24m45.592643083s) [upload=390995]: Got next segment. Inline: false Dec 24 04:53:45.721648986 (445h24m45.592732032s) [upload=390995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:53:45.721656201 (445h24m45.592739246s) [upload=390995]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:53:45.968344117 (445h24m45.839427152s) [seg_pos={0 0}, upload=390995]: Begin upload segment... Dec 24 04:53:45.968399892 (445h24m45.839482927s) [seg_pos={0 0}, upload=390995]: Done begin upload segment. Dec 24 04:53:45.968504732 (445h24m45.839587767s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.968571449 (445h24m45.839654484s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969513152 (445h24m45.840596187s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969756015 (445h24m45.84083905s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970245246 (445h24m45.841328281s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970597858 (445h24m45.841680903s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970970517 (445h24m45.842053553s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971362764 (445h24m45.842445799s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971712941 (445h24m45.842795986s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972101662 (445h24m45.843184697s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972560385 (445h24m45.84364342s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.974902435 (445h24m45.84598547s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.975744338 (445h24m45.846827374s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.976475501 (445h24m45.847558546s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.977774053 (445h24m45.848857098s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.978307899 (445h24m45.849390934s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:47.684950330 (445h24m47.556033365s) [upload=390995]: Next returned nil segment Dec 24 04:53:47.684967432 (445h24m47.556050477s) [upload=390995]: Waiting for error group managing segments... Dec 24 04:53:47.975268627 (445h24m47.846351662s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.976768513 (445h24m47.847851558s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.977666483 (445h24m47.848749528s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.979492480 (445h24m47.850575515s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.982702883 (445h24m47.853785919s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.990496404 (445h24m47.861579449s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.019123966 (445h24m47.890207001s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.205478756 (445h24m48.076561801s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.215568068 (445h24m48.086651114s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.255055000 (445h24m48.126138045s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.360364752 (445h24m48.231447787s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:14.990184584 (445h25m14.861267629s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:15.032082447 (445h25m14.903165492s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Dec 24 04:54:15.032375033 (445h25m14.903458078s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection Dec 24 04:54:15.032605533 (445h25m14.903688578s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, address:145.79.227.50:4375): protocol: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection Dec 24 04:54:15.032711815 (445h25m14.90379486s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection Dec 24 04:54:15.032736873 (445h25m14.903819908s) [seg_pos={0 0}, upload=390995]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:54:15.278462246 (445h25m15.149545291s) [upload=390995]: Done uploading segments... Dec 24 04:54:15.278513083 (445h25m15.149596138s) [upload=390995]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 20.277382187s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X..............X..................] duration: 5.678568689s trace: https://jaeger.dev.storj.io/trace/1deda862aa4073e2 FAIL (2025-12-24 05:08:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:53:45.720403184 (445h24m45.591486219s) [upload=390995]: Starting upload Dec 24 04:53:45.720540295 (445h24m45.59162335s) [upload=390995]: Done starting upload Dec 24 04:53:45.720783568 (445h24m45.591866603s) [upload=390995]: Uploading segments... Dec 24 04:53:45.721560018 (445h24m45.592643083s) [upload=390995]: Got next segment. Inline: false Dec 24 04:53:45.721648986 (445h24m45.592732032s) [upload=390995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:53:45.721656201 (445h24m45.592739246s) [upload=390995]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:53:45.968344117 (445h24m45.839427152s) [seg_pos={0 0}, upload=390995]: Begin upload segment... Dec 24 04:53:45.968399892 (445h24m45.839482927s) [seg_pos={0 0}, upload=390995]: Done begin upload segment. Dec 24 04:53:45.968504732 (445h24m45.839587767s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.968571449 (445h24m45.839654484s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969513152 (445h24m45.840596187s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969756015 (445h24m45.84083905s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970245246 (445h24m45.841328281s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970597858 (445h24m45.841680903s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970970517 (445h24m45.842053553s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971362764 (445h24m45.842445799s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971712941 (445h24m45.842795986s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972101662 (445h24m45.843184697s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972560385 (445h24m45.84364342s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.974902435 (445h24m45.84598547s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.975744338 (445h24m45.846827374s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.976475501 (445h24m45.847558546s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.977774053 (445h24m45.848857098s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.978307899 (445h24m45.849390934s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:47.684950330 (445h24m47.556033365s) [upload=390995]: Next returned nil segment Dec 24 04:53:47.684967432 (445h24m47.556050477s) [upload=390995]: Waiting for error group managing segments... Dec 24 04:53:47.975268627 (445h24m47.846351662s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.976768513 (445h24m47.847851558s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.977666483 (445h24m47.848749528s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.979492480 (445h24m47.850575515s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.982702883 (445h24m47.853785919s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.990496404 (445h24m47.861579449s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.019123966 (445h24m47.890207001s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.205478756 (445h24m48.076561801s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.215568068 (445h24m48.086651114s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.255055000 (445h24m48.126138045s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.360364752 (445h24m48.231447787s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:14.990184584 (445h25m14.861267629s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:15.032082447 (445h25m14.903165492s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Dec 24 04:54:15.032375033 (445h25m14.903458078s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection Dec 24 04:54:15.032605533 (445h25m14.903688578s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, address:145.79.227.50:4375): protocol: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection Dec 24 04:54:15.032711815 (445h25m14.90379486s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection Dec 24 04:54:15.032736873 (445h25m14.903819908s) [seg_pos={0 0}, upload=390995]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:54:15.278462246 (445h25m15.149545291s) [upload=390995]: Done uploading segments... Dec 24 04:54:15.278513083 (445h25m15.149596138s) [upload=390995]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.678592965s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X..............X.................] duration: 5.854168963s trace: https://jaeger.dev.storj.io/trace/1deda862aa4073e2 FAIL (2025-12-24 05:08:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:53:45.720403184 (445h24m45.591486219s) [upload=390995]: Starting upload Dec 24 04:53:45.720540295 (445h24m45.59162335s) [upload=390995]: Done starting upload Dec 24 04:53:45.720783568 (445h24m45.591866603s) [upload=390995]: Uploading segments... Dec 24 04:53:45.721560018 (445h24m45.592643083s) [upload=390995]: Got next segment. Inline: false Dec 24 04:53:45.721648986 (445h24m45.592732032s) [upload=390995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:53:45.721656201 (445h24m45.592739246s) [upload=390995]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:53:45.968344117 (445h24m45.839427152s) [seg_pos={0 0}, upload=390995]: Begin upload segment... Dec 24 04:53:45.968399892 (445h24m45.839482927s) [seg_pos={0 0}, upload=390995]: Done begin upload segment. Dec 24 04:53:45.968504732 (445h24m45.839587767s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.968571449 (445h24m45.839654484s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969513152 (445h24m45.840596187s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969756015 (445h24m45.84083905s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970245246 (445h24m45.841328281s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970597858 (445h24m45.841680903s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970970517 (445h24m45.842053553s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971362764 (445h24m45.842445799s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971712941 (445h24m45.842795986s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972101662 (445h24m45.843184697s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972560385 (445h24m45.84364342s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.974902435 (445h24m45.84598547s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.975744338 (445h24m45.846827374s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.976475501 (445h24m45.847558546s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.977774053 (445h24m45.848857098s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.978307899 (445h24m45.849390934s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:47.684950330 (445h24m47.556033365s) [upload=390995]: Next returned nil segment Dec 24 04:53:47.684967432 (445h24m47.556050477s) [upload=390995]: Waiting for error group managing segments... Dec 24 04:53:47.975268627 (445h24m47.846351662s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.976768513 (445h24m47.847851558s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.977666483 (445h24m47.848749528s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.979492480 (445h24m47.850575515s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.982702883 (445h24m47.853785919s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.990496404 (445h24m47.861579449s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.019123966 (445h24m47.890207001s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.205478756 (445h24m48.076561801s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.215568068 (445h24m48.086651114s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.255055000 (445h24m48.126138045s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.360364752 (445h24m48.231447787s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:14.990184584 (445h25m14.861267629s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:15.032082447 (445h25m14.903165492s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Dec 24 04:54:15.032375033 (445h25m14.903458078s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection Dec 24 04:54:15.032605533 (445h25m14.903688578s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, address:145.79.227.50:4375): protocol: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection Dec 24 04:54:15.032711815 (445h25m14.90379486s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection Dec 24 04:54:15.032736873 (445h25m14.903819908s) [seg_pos={0 0}, upload=390995]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:54:15.278462246 (445h25m15.149545291s) [upload=390995]: Done uploading segments... Dec 24 04:54:15.278513083 (445h25m15.149596138s) [upload=390995]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.854206034s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X..............X................] duration: 5.91838405s trace: https://jaeger.dev.storj.io/trace/1deda862aa4073e2 FAIL (2025-12-24 05:07:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:53:45.720403184 (445h24m45.591486219s) [upload=390995]: Starting upload Dec 24 04:53:45.720540295 (445h24m45.59162335s) [upload=390995]: Done starting upload Dec 24 04:53:45.720783568 (445h24m45.591866603s) [upload=390995]: Uploading segments... Dec 24 04:53:45.721560018 (445h24m45.592643083s) [upload=390995]: Got next segment. Inline: false Dec 24 04:53:45.721648986 (445h24m45.592732032s) [upload=390995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:53:45.721656201 (445h24m45.592739246s) [upload=390995]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:53:45.968344117 (445h24m45.839427152s) [seg_pos={0 0}, upload=390995]: Begin upload segment... Dec 24 04:53:45.968399892 (445h24m45.839482927s) [seg_pos={0 0}, upload=390995]: Done begin upload segment. Dec 24 04:53:45.968504732 (445h24m45.839587767s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.968571449 (445h24m45.839654484s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969513152 (445h24m45.840596187s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969756015 (445h24m45.84083905s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970245246 (445h24m45.841328281s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970597858 (445h24m45.841680903s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970970517 (445h24m45.842053553s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971362764 (445h24m45.842445799s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971712941 (445h24m45.842795986s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972101662 (445h24m45.843184697s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972560385 (445h24m45.84364342s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.974902435 (445h24m45.84598547s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.975744338 (445h24m45.846827374s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.976475501 (445h24m45.847558546s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.977774053 (445h24m45.848857098s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.978307899 (445h24m45.849390934s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:47.684950330 (445h24m47.556033365s) [upload=390995]: Next returned nil segment Dec 24 04:53:47.684967432 (445h24m47.556050477s) [upload=390995]: Waiting for error group managing segments... Dec 24 04:53:47.975268627 (445h24m47.846351662s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.976768513 (445h24m47.847851558s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.977666483 (445h24m47.848749528s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.979492480 (445h24m47.850575515s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.982702883 (445h24m47.853785919s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.990496404 (445h24m47.861579449s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.019123966 (445h24m47.890207001s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.205478756 (445h24m48.076561801s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.215568068 (445h24m48.086651114s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.255055000 (445h24m48.126138045s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.360364752 (445h24m48.231447787s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:14.990184584 (445h25m14.861267629s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:15.032082447 (445h25m14.903165492s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Dec 24 04:54:15.032375033 (445h25m14.903458078s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection Dec 24 04:54:15.032605533 (445h25m14.903688578s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, address:145.79.227.50:4375): protocol: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection Dec 24 04:54:15.032711815 (445h25m14.90379486s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection Dec 24 04:54:15.032736873 (445h25m14.903819908s) [seg_pos={0 0}, upload=390995]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:54:15.278462246 (445h25m15.149545291s) [upload=390995]: Done uploading segments... Dec 24 04:54:15.278513083 (445h25m15.149596138s) [upload=390995]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.918411954s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X..............X...............] duration: 5.876783469s trace: https://jaeger.dev.storj.io/trace/1deda862aa4073e2 FAIL (2025-12-24 05:06:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:53:45.720403184 (445h24m45.591486219s) [upload=390995]: Starting upload Dec 24 04:53:45.720540295 (445h24m45.59162335s) [upload=390995]: Done starting upload Dec 24 04:53:45.720783568 (445h24m45.591866603s) [upload=390995]: Uploading segments... Dec 24 04:53:45.721560018 (445h24m45.592643083s) [upload=390995]: Got next segment. Inline: false Dec 24 04:53:45.721648986 (445h24m45.592732032s) [upload=390995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:53:45.721656201 (445h24m45.592739246s) [upload=390995]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:53:45.968344117 (445h24m45.839427152s) [seg_pos={0 0}, upload=390995]: Begin upload segment... Dec 24 04:53:45.968399892 (445h24m45.839482927s) [seg_pos={0 0}, upload=390995]: Done begin upload segment. Dec 24 04:53:45.968504732 (445h24m45.839587767s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.968571449 (445h24m45.839654484s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969513152 (445h24m45.840596187s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969756015 (445h24m45.84083905s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970245246 (445h24m45.841328281s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970597858 (445h24m45.841680903s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970970517 (445h24m45.842053553s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971362764 (445h24m45.842445799s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971712941 (445h24m45.842795986s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972101662 (445h24m45.843184697s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972560385 (445h24m45.84364342s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.974902435 (445h24m45.84598547s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.975744338 (445h24m45.846827374s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.976475501 (445h24m45.847558546s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.977774053 (445h24m45.848857098s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.978307899 (445h24m45.849390934s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:47.684950330 (445h24m47.556033365s) [upload=390995]: Next returned nil segment Dec 24 04:53:47.684967432 (445h24m47.556050477s) [upload=390995]: Waiting for error group managing segments... Dec 24 04:53:47.975268627 (445h24m47.846351662s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.976768513 (445h24m47.847851558s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.977666483 (445h24m47.848749528s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.979492480 (445h24m47.850575515s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.982702883 (445h24m47.853785919s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.990496404 (445h24m47.861579449s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.019123966 (445h24m47.890207001s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.205478756 (445h24m48.076561801s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.215568068 (445h24m48.086651114s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.255055000 (445h24m48.126138045s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.360364752 (445h24m48.231447787s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:14.990184584 (445h25m14.861267629s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:15.032082447 (445h25m14.903165492s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Dec 24 04:54:15.032375033 (445h25m14.903458078s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection Dec 24 04:54:15.032605533 (445h25m14.903688578s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, address:145.79.227.50:4375): protocol: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection Dec 24 04:54:15.032711815 (445h25m14.90379486s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection Dec 24 04:54:15.032736873 (445h25m14.903819908s) [seg_pos={0 0}, upload=390995]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:54:15.278462246 (445h25m15.149545291s) [upload=390995]: Done uploading segments... Dec 24 04:54:15.278513083 (445h25m15.149596138s) [upload=390995]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.876811753s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X..............X..............] duration: 6.330198859s trace: https://jaeger.dev.storj.io/trace/1deda862aa4073e2 FAIL (2025-12-24 05:06:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:53:45.720403184 (445h24m45.591486219s) [upload=390995]: Starting upload Dec 24 04:53:45.720540295 (445h24m45.59162335s) [upload=390995]: Done starting upload Dec 24 04:53:45.720783568 (445h24m45.591866603s) [upload=390995]: Uploading segments... Dec 24 04:53:45.721560018 (445h24m45.592643083s) [upload=390995]: Got next segment. Inline: false Dec 24 04:53:45.721648986 (445h24m45.592732032s) [upload=390995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:53:45.721656201 (445h24m45.592739246s) [upload=390995]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:53:45.968344117 (445h24m45.839427152s) [seg_pos={0 0}, upload=390995]: Begin upload segment... Dec 24 04:53:45.968399892 (445h24m45.839482927s) [seg_pos={0 0}, upload=390995]: Done begin upload segment. Dec 24 04:53:45.968504732 (445h24m45.839587767s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.968571449 (445h24m45.839654484s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969513152 (445h24m45.840596187s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969756015 (445h24m45.84083905s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970245246 (445h24m45.841328281s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970597858 (445h24m45.841680903s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970970517 (445h24m45.842053553s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971362764 (445h24m45.842445799s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971712941 (445h24m45.842795986s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972101662 (445h24m45.843184697s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972560385 (445h24m45.84364342s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.974902435 (445h24m45.84598547s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.975744338 (445h24m45.846827374s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.976475501 (445h24m45.847558546s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.977774053 (445h24m45.848857098s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.978307899 (445h24m45.849390934s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:47.684950330 (445h24m47.556033365s) [upload=390995]: Next returned nil segment Dec 24 04:53:47.684967432 (445h24m47.556050477s) [upload=390995]: Waiting for error group managing segments... Dec 24 04:53:47.975268627 (445h24m47.846351662s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.976768513 (445h24m47.847851558s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.977666483 (445h24m47.848749528s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.979492480 (445h24m47.850575515s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.982702883 (445h24m47.853785919s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.990496404 (445h24m47.861579449s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.019123966 (445h24m47.890207001s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.205478756 (445h24m48.076561801s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.215568068 (445h24m48.086651114s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.255055000 (445h24m48.126138045s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.360364752 (445h24m48.231447787s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:14.990184584 (445h25m14.861267629s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:15.032082447 (445h25m14.903165492s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Dec 24 04:54:15.032375033 (445h25m14.903458078s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection Dec 24 04:54:15.032605533 (445h25m14.903688578s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, address:145.79.227.50:4375): protocol: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection Dec 24 04:54:15.032711815 (445h25m14.90379486s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection Dec 24 04:54:15.032736873 (445h25m14.903819908s) [seg_pos={0 0}, upload=390995]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:54:15.278462246 (445h25m15.149545291s) [upload=390995]: Done uploading segments... Dec 24 04:54:15.278513083 (445h25m15.149596138s) [upload=390995]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.330230378s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X..............X.............] duration: 5.872593576s trace: https://jaeger.dev.storj.io/trace/1deda862aa4073e2 FAIL (2025-12-24 05:05:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:53:45.720403184 (445h24m45.591486219s) [upload=390995]: Starting upload Dec 24 04:53:45.720540295 (445h24m45.59162335s) [upload=390995]: Done starting upload Dec 24 04:53:45.720783568 (445h24m45.591866603s) [upload=390995]: Uploading segments... Dec 24 04:53:45.721560018 (445h24m45.592643083s) [upload=390995]: Got next segment. Inline: false Dec 24 04:53:45.721648986 (445h24m45.592732032s) [upload=390995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:53:45.721656201 (445h24m45.592739246s) [upload=390995]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:53:45.968344117 (445h24m45.839427152s) [seg_pos={0 0}, upload=390995]: Begin upload segment... Dec 24 04:53:45.968399892 (445h24m45.839482927s) [seg_pos={0 0}, upload=390995]: Done begin upload segment. Dec 24 04:53:45.968504732 (445h24m45.839587767s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.968571449 (445h24m45.839654484s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969513152 (445h24m45.840596187s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969756015 (445h24m45.84083905s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970245246 (445h24m45.841328281s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970597858 (445h24m45.841680903s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970970517 (445h24m45.842053553s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971362764 (445h24m45.842445799s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971712941 (445h24m45.842795986s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972101662 (445h24m45.843184697s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972560385 (445h24m45.84364342s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.974902435 (445h24m45.84598547s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.975744338 (445h24m45.846827374s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.976475501 (445h24m45.847558546s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.977774053 (445h24m45.848857098s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.978307899 (445h24m45.849390934s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:47.684950330 (445h24m47.556033365s) [upload=390995]: Next returned nil segment Dec 24 04:53:47.684967432 (445h24m47.556050477s) [upload=390995]: Waiting for error group managing segments... Dec 24 04:53:47.975268627 (445h24m47.846351662s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.976768513 (445h24m47.847851558s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.977666483 (445h24m47.848749528s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.979492480 (445h24m47.850575515s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.982702883 (445h24m47.853785919s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.990496404 (445h24m47.861579449s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.019123966 (445h24m47.890207001s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.205478756 (445h24m48.076561801s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.215568068 (445h24m48.086651114s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.255055000 (445h24m48.126138045s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.360364752 (445h24m48.231447787s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:14.990184584 (445h25m14.861267629s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:15.032082447 (445h25m14.903165492s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Dec 24 04:54:15.032375033 (445h25m14.903458078s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection Dec 24 04:54:15.032605533 (445h25m14.903688578s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, address:145.79.227.50:4375): protocol: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection Dec 24 04:54:15.032711815 (445h25m14.90379486s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection Dec 24 04:54:15.032736873 (445h25m14.903819908s) [seg_pos={0 0}, upload=390995]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:54:15.278462246 (445h25m15.149545291s) [upload=390995]: Done uploading segments... Dec 24 04:54:15.278513083 (445h25m15.149596138s) [upload=390995]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.872615058s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X..............X............] duration: 5.913024146s trace: https://jaeger.dev.storj.io/trace/1deda862aa4073e2 FAIL (2025-12-24 05:04:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:53:45.720403184 (445h24m45.591486219s) [upload=390995]: Starting upload Dec 24 04:53:45.720540295 (445h24m45.59162335s) [upload=390995]: Done starting upload Dec 24 04:53:45.720783568 (445h24m45.591866603s) [upload=390995]: Uploading segments... Dec 24 04:53:45.721560018 (445h24m45.592643083s) [upload=390995]: Got next segment. Inline: false Dec 24 04:53:45.721648986 (445h24m45.592732032s) [upload=390995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:53:45.721656201 (445h24m45.592739246s) [upload=390995]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:53:45.968344117 (445h24m45.839427152s) [seg_pos={0 0}, upload=390995]: Begin upload segment... Dec 24 04:53:45.968399892 (445h24m45.839482927s) [seg_pos={0 0}, upload=390995]: Done begin upload segment. Dec 24 04:53:45.968504732 (445h24m45.839587767s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.968571449 (445h24m45.839654484s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969513152 (445h24m45.840596187s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969756015 (445h24m45.84083905s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970245246 (445h24m45.841328281s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970597858 (445h24m45.841680903s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970970517 (445h24m45.842053553s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971362764 (445h24m45.842445799s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971712941 (445h24m45.842795986s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972101662 (445h24m45.843184697s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972560385 (445h24m45.84364342s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.974902435 (445h24m45.84598547s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.975744338 (445h24m45.846827374s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.976475501 (445h24m45.847558546s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.977774053 (445h24m45.848857098s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.978307899 (445h24m45.849390934s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:47.684950330 (445h24m47.556033365s) [upload=390995]: Next returned nil segment Dec 24 04:53:47.684967432 (445h24m47.556050477s) [upload=390995]: Waiting for error group managing segments... Dec 24 04:53:47.975268627 (445h24m47.846351662s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.976768513 (445h24m47.847851558s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.977666483 (445h24m47.848749528s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.979492480 (445h24m47.850575515s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.982702883 (445h24m47.853785919s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.990496404 (445h24m47.861579449s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.019123966 (445h24m47.890207001s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.205478756 (445h24m48.076561801s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.215568068 (445h24m48.086651114s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.255055000 (445h24m48.126138045s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.360364752 (445h24m48.231447787s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:14.990184584 (445h25m14.861267629s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:15.032082447 (445h25m14.903165492s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Dec 24 04:54:15.032375033 (445h25m14.903458078s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection Dec 24 04:54:15.032605533 (445h25m14.903688578s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, address:145.79.227.50:4375): protocol: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection Dec 24 04:54:15.032711815 (445h25m14.90379486s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection Dec 24 04:54:15.032736873 (445h25m14.903819908s) [seg_pos={0 0}, upload=390995]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:54:15.278462246 (445h25m15.149545291s) [upload=390995]: Done uploading segments... Dec 24 04:54:15.278513083 (445h25m15.149596138s) [upload=390995]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.913050366s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X..............X...........] duration: 23.66272891s trace: https://jaeger.dev.storj.io/trace/1deda862aa4073e2 FAIL (2025-12-24 05:03:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:53:45.720403184 (445h24m45.591486219s) [upload=390995]: Starting upload Dec 24 04:53:45.720540295 (445h24m45.59162335s) [upload=390995]: Done starting upload Dec 24 04:53:45.720783568 (445h24m45.591866603s) [upload=390995]: Uploading segments... Dec 24 04:53:45.721560018 (445h24m45.592643083s) [upload=390995]: Got next segment. Inline: false Dec 24 04:53:45.721648986 (445h24m45.592732032s) [upload=390995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:53:45.721656201 (445h24m45.592739246s) [upload=390995]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:53:45.968344117 (445h24m45.839427152s) [seg_pos={0 0}, upload=390995]: Begin upload segment... Dec 24 04:53:45.968399892 (445h24m45.839482927s) [seg_pos={0 0}, upload=390995]: Done begin upload segment. Dec 24 04:53:45.968504732 (445h24m45.839587767s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.968571449 (445h24m45.839654484s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969513152 (445h24m45.840596187s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969756015 (445h24m45.84083905s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970245246 (445h24m45.841328281s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970597858 (445h24m45.841680903s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970970517 (445h24m45.842053553s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971362764 (445h24m45.842445799s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971712941 (445h24m45.842795986s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972101662 (445h24m45.843184697s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972560385 (445h24m45.84364342s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.974902435 (445h24m45.84598547s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.975744338 (445h24m45.846827374s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.976475501 (445h24m45.847558546s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.977774053 (445h24m45.848857098s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.978307899 (445h24m45.849390934s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:47.684950330 (445h24m47.556033365s) [upload=390995]: Next returned nil segment Dec 24 04:53:47.684967432 (445h24m47.556050477s) [upload=390995]: Waiting for error group managing segments... Dec 24 04:53:47.975268627 (445h24m47.846351662s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.976768513 (445h24m47.847851558s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.977666483 (445h24m47.848749528s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.979492480 (445h24m47.850575515s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.982702883 (445h24m47.853785919s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.990496404 (445h24m47.861579449s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.019123966 (445h24m47.890207001s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.205478756 (445h24m48.076561801s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.215568068 (445h24m48.086651114s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.255055000 (445h24m48.126138045s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.360364752 (445h24m48.231447787s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:14.990184584 (445h25m14.861267629s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:15.032082447 (445h25m14.903165492s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Dec 24 04:54:15.032375033 (445h25m14.903458078s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection Dec 24 04:54:15.032605533 (445h25m14.903688578s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, address:145.79.227.50:4375): protocol: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection Dec 24 04:54:15.032711815 (445h25m14.90379486s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection Dec 24 04:54:15.032736873 (445h25m14.903819908s) [seg_pos={0 0}, upload=390995]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:54:15.278462246 (445h25m15.149545291s) [upload=390995]: Done uploading segments... Dec 24 04:54:15.278513083 (445h25m15.149596138s) [upload=390995]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 23.662758867s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X..............X..........] duration: 5.684810456s trace: https://jaeger.dev.storj.io/trace/1deda862aa4073e2 FAIL (2025-12-24 05:02:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:53:45.720403184 (445h24m45.591486219s) [upload=390995]: Starting upload Dec 24 04:53:45.720540295 (445h24m45.59162335s) [upload=390995]: Done starting upload Dec 24 04:53:45.720783568 (445h24m45.591866603s) [upload=390995]: Uploading segments... Dec 24 04:53:45.721560018 (445h24m45.592643083s) [upload=390995]: Got next segment. Inline: false Dec 24 04:53:45.721648986 (445h24m45.592732032s) [upload=390995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:53:45.721656201 (445h24m45.592739246s) [upload=390995]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:53:45.968344117 (445h24m45.839427152s) [seg_pos={0 0}, upload=390995]: Begin upload segment... Dec 24 04:53:45.968399892 (445h24m45.839482927s) [seg_pos={0 0}, upload=390995]: Done begin upload segment. Dec 24 04:53:45.968504732 (445h24m45.839587767s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.968571449 (445h24m45.839654484s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969513152 (445h24m45.840596187s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969756015 (445h24m45.84083905s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970245246 (445h24m45.841328281s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970597858 (445h24m45.841680903s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970970517 (445h24m45.842053553s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971362764 (445h24m45.842445799s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971712941 (445h24m45.842795986s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972101662 (445h24m45.843184697s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972560385 (445h24m45.84364342s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.974902435 (445h24m45.84598547s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.975744338 (445h24m45.846827374s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.976475501 (445h24m45.847558546s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.977774053 (445h24m45.848857098s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.978307899 (445h24m45.849390934s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:47.684950330 (445h24m47.556033365s) [upload=390995]: Next returned nil segment Dec 24 04:53:47.684967432 (445h24m47.556050477s) [upload=390995]: Waiting for error group managing segments... Dec 24 04:53:47.975268627 (445h24m47.846351662s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.976768513 (445h24m47.847851558s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.977666483 (445h24m47.848749528s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.979492480 (445h24m47.850575515s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.982702883 (445h24m47.853785919s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.990496404 (445h24m47.861579449s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.019123966 (445h24m47.890207001s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.205478756 (445h24m48.076561801s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.215568068 (445h24m48.086651114s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.255055000 (445h24m48.126138045s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.360364752 (445h24m48.231447787s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:14.990184584 (445h25m14.861267629s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:15.032082447 (445h25m14.903165492s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Dec 24 04:54:15.032375033 (445h25m14.903458078s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection Dec 24 04:54:15.032605533 (445h25m14.903688578s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, address:145.79.227.50:4375): protocol: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection Dec 24 04:54:15.032711815 (445h25m14.90379486s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection Dec 24 04:54:15.032736873 (445h25m14.903819908s) [seg_pos={0 0}, upload=390995]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:54:15.278462246 (445h25m15.149545291s) [upload=390995]: Done uploading segments... Dec 24 04:54:15.278513083 (445h25m15.149596138s) [upload=390995]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.684840252s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X..............X.........] duration: 5.70823439s trace: https://jaeger.dev.storj.io/trace/1deda862aa4073e2 FAIL (2025-12-24 05:02:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:53:45.720403184 (445h24m45.591486219s) [upload=390995]: Starting upload Dec 24 04:53:45.720540295 (445h24m45.59162335s) [upload=390995]: Done starting upload Dec 24 04:53:45.720783568 (445h24m45.591866603s) [upload=390995]: Uploading segments... Dec 24 04:53:45.721560018 (445h24m45.592643083s) [upload=390995]: Got next segment. Inline: false Dec 24 04:53:45.721648986 (445h24m45.592732032s) [upload=390995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:53:45.721656201 (445h24m45.592739246s) [upload=390995]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:53:45.968344117 (445h24m45.839427152s) [seg_pos={0 0}, upload=390995]: Begin upload segment... Dec 24 04:53:45.968399892 (445h24m45.839482927s) [seg_pos={0 0}, upload=390995]: Done begin upload segment. Dec 24 04:53:45.968504732 (445h24m45.839587767s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.968571449 (445h24m45.839654484s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969513152 (445h24m45.840596187s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969756015 (445h24m45.84083905s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970245246 (445h24m45.841328281s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970597858 (445h24m45.841680903s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970970517 (445h24m45.842053553s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971362764 (445h24m45.842445799s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971712941 (445h24m45.842795986s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972101662 (445h24m45.843184697s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972560385 (445h24m45.84364342s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.974902435 (445h24m45.84598547s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.975744338 (445h24m45.846827374s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.976475501 (445h24m45.847558546s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.977774053 (445h24m45.848857098s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.978307899 (445h24m45.849390934s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:47.684950330 (445h24m47.556033365s) [upload=390995]: Next returned nil segment Dec 24 04:53:47.684967432 (445h24m47.556050477s) [upload=390995]: Waiting for error group managing segments... Dec 24 04:53:47.975268627 (445h24m47.846351662s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.976768513 (445h24m47.847851558s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.977666483 (445h24m47.848749528s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.979492480 (445h24m47.850575515s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.982702883 (445h24m47.853785919s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.990496404 (445h24m47.861579449s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.019123966 (445h24m47.890207001s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.205478756 (445h24m48.076561801s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.215568068 (445h24m48.086651114s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.255055000 (445h24m48.126138045s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.360364752 (445h24m48.231447787s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:14.990184584 (445h25m14.861267629s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:15.032082447 (445h25m14.903165492s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Dec 24 04:54:15.032375033 (445h25m14.903458078s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection Dec 24 04:54:15.032605533 (445h25m14.903688578s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, address:145.79.227.50:4375): protocol: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection Dec 24 04:54:15.032711815 (445h25m14.90379486s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection Dec 24 04:54:15.032736873 (445h25m14.903819908s) [seg_pos={0 0}, upload=390995]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:54:15.278462246 (445h25m15.149545291s) [upload=390995]: Done uploading segments... Dec 24 04:54:15.278513083 (445h25m15.149596138s) [upload=390995]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.708260882s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X..............X........] duration: 6.002789259s trace: https://jaeger.dev.storj.io/trace/1deda862aa4073e2 FAIL (2025-12-24 05:01:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:53:45.720403184 (445h24m45.591486219s) [upload=390995]: Starting upload Dec 24 04:53:45.720540295 (445h24m45.59162335s) [upload=390995]: Done starting upload Dec 24 04:53:45.720783568 (445h24m45.591866603s) [upload=390995]: Uploading segments... Dec 24 04:53:45.721560018 (445h24m45.592643083s) [upload=390995]: Got next segment. Inline: false Dec 24 04:53:45.721648986 (445h24m45.592732032s) [upload=390995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:53:45.721656201 (445h24m45.592739246s) [upload=390995]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:53:45.968344117 (445h24m45.839427152s) [seg_pos={0 0}, upload=390995]: Begin upload segment... Dec 24 04:53:45.968399892 (445h24m45.839482927s) [seg_pos={0 0}, upload=390995]: Done begin upload segment. Dec 24 04:53:45.968504732 (445h24m45.839587767s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.968571449 (445h24m45.839654484s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969513152 (445h24m45.840596187s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969756015 (445h24m45.84083905s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970245246 (445h24m45.841328281s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970597858 (445h24m45.841680903s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970970517 (445h24m45.842053553s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971362764 (445h24m45.842445799s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971712941 (445h24m45.842795986s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972101662 (445h24m45.843184697s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972560385 (445h24m45.84364342s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.974902435 (445h24m45.84598547s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.975744338 (445h24m45.846827374s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.976475501 (445h24m45.847558546s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.977774053 (445h24m45.848857098s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.978307899 (445h24m45.849390934s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:47.684950330 (445h24m47.556033365s) [upload=390995]: Next returned nil segment Dec 24 04:53:47.684967432 (445h24m47.556050477s) [upload=390995]: Waiting for error group managing segments... Dec 24 04:53:47.975268627 (445h24m47.846351662s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.976768513 (445h24m47.847851558s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.977666483 (445h24m47.848749528s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.979492480 (445h24m47.850575515s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.982702883 (445h24m47.853785919s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.990496404 (445h24m47.861579449s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.019123966 (445h24m47.890207001s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.205478756 (445h24m48.076561801s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.215568068 (445h24m48.086651114s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.255055000 (445h24m48.126138045s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.360364752 (445h24m48.231447787s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:14.990184584 (445h25m14.861267629s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:15.032082447 (445h25m14.903165492s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Dec 24 04:54:15.032375033 (445h25m14.903458078s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection Dec 24 04:54:15.032605533 (445h25m14.903688578s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, address:145.79.227.50:4375): protocol: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection Dec 24 04:54:15.032711815 (445h25m14.90379486s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection Dec 24 04:54:15.032736873 (445h25m14.903819908s) [seg_pos={0 0}, upload=390995]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:54:15.278462246 (445h25m15.149545291s) [upload=390995]: Done uploading segments... Dec 24 04:54:15.278513083 (445h25m15.149596138s) [upload=390995]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.002851408s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X..............X.......] duration: 5.699561536s trace: https://jaeger.dev.storj.io/trace/1deda862aa4073e2 FAIL (2025-12-24 05:00:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:53:45.720403184 (445h24m45.591486219s) [upload=390995]: Starting upload Dec 24 04:53:45.720540295 (445h24m45.59162335s) [upload=390995]: Done starting upload Dec 24 04:53:45.720783568 (445h24m45.591866603s) [upload=390995]: Uploading segments... Dec 24 04:53:45.721560018 (445h24m45.592643083s) [upload=390995]: Got next segment. Inline: false Dec 24 04:53:45.721648986 (445h24m45.592732032s) [upload=390995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:53:45.721656201 (445h24m45.592739246s) [upload=390995]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:53:45.968344117 (445h24m45.839427152s) [seg_pos={0 0}, upload=390995]: Begin upload segment... Dec 24 04:53:45.968399892 (445h24m45.839482927s) [seg_pos={0 0}, upload=390995]: Done begin upload segment. Dec 24 04:53:45.968504732 (445h24m45.839587767s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.968571449 (445h24m45.839654484s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969513152 (445h24m45.840596187s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969756015 (445h24m45.84083905s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970245246 (445h24m45.841328281s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970597858 (445h24m45.841680903s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970970517 (445h24m45.842053553s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971362764 (445h24m45.842445799s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971712941 (445h24m45.842795986s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972101662 (445h24m45.843184697s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972560385 (445h24m45.84364342s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.974902435 (445h24m45.84598547s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.975744338 (445h24m45.846827374s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.976475501 (445h24m45.847558546s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.977774053 (445h24m45.848857098s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.978307899 (445h24m45.849390934s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:47.684950330 (445h24m47.556033365s) [upload=390995]: Next returned nil segment Dec 24 04:53:47.684967432 (445h24m47.556050477s) [upload=390995]: Waiting for error group managing segments... Dec 24 04:53:47.975268627 (445h24m47.846351662s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.976768513 (445h24m47.847851558s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.977666483 (445h24m47.848749528s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.979492480 (445h24m47.850575515s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.982702883 (445h24m47.853785919s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.990496404 (445h24m47.861579449s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.019123966 (445h24m47.890207001s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.205478756 (445h24m48.076561801s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.215568068 (445h24m48.086651114s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.255055000 (445h24m48.126138045s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.360364752 (445h24m48.231447787s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:14.990184584 (445h25m14.861267629s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:15.032082447 (445h25m14.903165492s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Dec 24 04:54:15.032375033 (445h25m14.903458078s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection Dec 24 04:54:15.032605533 (445h25m14.903688578s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, address:145.79.227.50:4375): protocol: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection Dec 24 04:54:15.032711815 (445h25m14.90379486s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection Dec 24 04:54:15.032736873 (445h25m14.903819908s) [seg_pos={0 0}, upload=390995]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:54:15.278462246 (445h25m15.149545291s) [upload=390995]: Done uploading segments... Dec 24 04:54:15.278513083 (445h25m15.149596138s) [upload=390995]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.699606561s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X..............X......] duration: 5.76998984s trace: https://jaeger.dev.storj.io/trace/1deda862aa4073e2 FAIL (2025-12-24 04:59:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:53:45.720403184 (445h24m45.591486219s) [upload=390995]: Starting upload Dec 24 04:53:45.720540295 (445h24m45.59162335s) [upload=390995]: Done starting upload Dec 24 04:53:45.720783568 (445h24m45.591866603s) [upload=390995]: Uploading segments... Dec 24 04:53:45.721560018 (445h24m45.592643083s) [upload=390995]: Got next segment. Inline: false Dec 24 04:53:45.721648986 (445h24m45.592732032s) [upload=390995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:53:45.721656201 (445h24m45.592739246s) [upload=390995]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:53:45.968344117 (445h24m45.839427152s) [seg_pos={0 0}, upload=390995]: Begin upload segment... Dec 24 04:53:45.968399892 (445h24m45.839482927s) [seg_pos={0 0}, upload=390995]: Done begin upload segment. Dec 24 04:53:45.968504732 (445h24m45.839587767s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.968571449 (445h24m45.839654484s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969513152 (445h24m45.840596187s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969756015 (445h24m45.84083905s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970245246 (445h24m45.841328281s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970597858 (445h24m45.841680903s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970970517 (445h24m45.842053553s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971362764 (445h24m45.842445799s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971712941 (445h24m45.842795986s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972101662 (445h24m45.843184697s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972560385 (445h24m45.84364342s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.974902435 (445h24m45.84598547s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.975744338 (445h24m45.846827374s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.976475501 (445h24m45.847558546s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.977774053 (445h24m45.848857098s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.978307899 (445h24m45.849390934s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:47.684950330 (445h24m47.556033365s) [upload=390995]: Next returned nil segment Dec 24 04:53:47.684967432 (445h24m47.556050477s) [upload=390995]: Waiting for error group managing segments... Dec 24 04:53:47.975268627 (445h24m47.846351662s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.976768513 (445h24m47.847851558s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.977666483 (445h24m47.848749528s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.979492480 (445h24m47.850575515s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.982702883 (445h24m47.853785919s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.990496404 (445h24m47.861579449s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.019123966 (445h24m47.890207001s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.205478756 (445h24m48.076561801s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.215568068 (445h24m48.086651114s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.255055000 (445h24m48.126138045s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.360364752 (445h24m48.231447787s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:14.990184584 (445h25m14.861267629s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:15.032082447 (445h25m14.903165492s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Dec 24 04:54:15.032375033 (445h25m14.903458078s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection Dec 24 04:54:15.032605533 (445h25m14.903688578s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, address:145.79.227.50:4375): protocol: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection Dec 24 04:54:15.032711815 (445h25m14.90379486s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection Dec 24 04:54:15.032736873 (445h25m14.903819908s) [seg_pos={0 0}, upload=390995]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:54:15.278462246 (445h25m15.149545291s) [upload=390995]: Done uploading segments... Dec 24 04:54:15.278513083 (445h25m15.149596138s) [upload=390995]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.770022592s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X..............X.....] duration: 5.724581369s trace: https://jaeger.dev.storj.io/trace/1deda862aa4073e2 FAIL (2025-12-24 04:58:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:53:45.720403184 (445h24m45.591486219s) [upload=390995]: Starting upload Dec 24 04:53:45.720540295 (445h24m45.59162335s) [upload=390995]: Done starting upload Dec 24 04:53:45.720783568 (445h24m45.591866603s) [upload=390995]: Uploading segments... Dec 24 04:53:45.721560018 (445h24m45.592643083s) [upload=390995]: Got next segment. Inline: false Dec 24 04:53:45.721648986 (445h24m45.592732032s) [upload=390995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:53:45.721656201 (445h24m45.592739246s) [upload=390995]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:53:45.968344117 (445h24m45.839427152s) [seg_pos={0 0}, upload=390995]: Begin upload segment... Dec 24 04:53:45.968399892 (445h24m45.839482927s) [seg_pos={0 0}, upload=390995]: Done begin upload segment. Dec 24 04:53:45.968504732 (445h24m45.839587767s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.968571449 (445h24m45.839654484s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969513152 (445h24m45.840596187s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969756015 (445h24m45.84083905s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970245246 (445h24m45.841328281s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970597858 (445h24m45.841680903s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970970517 (445h24m45.842053553s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971362764 (445h24m45.842445799s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971712941 (445h24m45.842795986s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972101662 (445h24m45.843184697s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972560385 (445h24m45.84364342s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.974902435 (445h24m45.84598547s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.975744338 (445h24m45.846827374s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.976475501 (445h24m45.847558546s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.977774053 (445h24m45.848857098s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.978307899 (445h24m45.849390934s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:47.684950330 (445h24m47.556033365s) [upload=390995]: Next returned nil segment Dec 24 04:53:47.684967432 (445h24m47.556050477s) [upload=390995]: Waiting for error group managing segments... Dec 24 04:53:47.975268627 (445h24m47.846351662s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.976768513 (445h24m47.847851558s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.977666483 (445h24m47.848749528s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.979492480 (445h24m47.850575515s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.982702883 (445h24m47.853785919s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.990496404 (445h24m47.861579449s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.019123966 (445h24m47.890207001s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.205478756 (445h24m48.076561801s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.215568068 (445h24m48.086651114s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.255055000 (445h24m48.126138045s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.360364752 (445h24m48.231447787s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:14.990184584 (445h25m14.861267629s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:15.032082447 (445h25m14.903165492s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Dec 24 04:54:15.032375033 (445h25m14.903458078s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection Dec 24 04:54:15.032605533 (445h25m14.903688578s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, address:145.79.227.50:4375): protocol: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection Dec 24 04:54:15.032711815 (445h25m14.90379486s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection Dec 24 04:54:15.032736873 (445h25m14.903819908s) [seg_pos={0 0}, upload=390995]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:54:15.278462246 (445h25m15.149545291s) [upload=390995]: Done uploading segments... Dec 24 04:54:15.278513083 (445h25m15.149596138s) [upload=390995]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.724611225s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................X..............X....] duration: 5.637211356s trace: https://jaeger.dev.storj.io/trace/1deda862aa4073e2 FAIL (2025-12-24 04:57:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:53:45.720403184 (445h24m45.591486219s) [upload=390995]: Starting upload Dec 24 04:53:45.720540295 (445h24m45.59162335s) [upload=390995]: Done starting upload Dec 24 04:53:45.720783568 (445h24m45.591866603s) [upload=390995]: Uploading segments... Dec 24 04:53:45.721560018 (445h24m45.592643083s) [upload=390995]: Got next segment. Inline: false Dec 24 04:53:45.721648986 (445h24m45.592732032s) [upload=390995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:53:45.721656201 (445h24m45.592739246s) [upload=390995]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:53:45.968344117 (445h24m45.839427152s) [seg_pos={0 0}, upload=390995]: Begin upload segment... Dec 24 04:53:45.968399892 (445h24m45.839482927s) [seg_pos={0 0}, upload=390995]: Done begin upload segment. Dec 24 04:53:45.968504732 (445h24m45.839587767s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.968571449 (445h24m45.839654484s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969513152 (445h24m45.840596187s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969756015 (445h24m45.84083905s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970245246 (445h24m45.841328281s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970597858 (445h24m45.841680903s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970970517 (445h24m45.842053553s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971362764 (445h24m45.842445799s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971712941 (445h24m45.842795986s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972101662 (445h24m45.843184697s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972560385 (445h24m45.84364342s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.974902435 (445h24m45.84598547s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.975744338 (445h24m45.846827374s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.976475501 (445h24m45.847558546s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.977774053 (445h24m45.848857098s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.978307899 (445h24m45.849390934s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:47.684950330 (445h24m47.556033365s) [upload=390995]: Next returned nil segment Dec 24 04:53:47.684967432 (445h24m47.556050477s) [upload=390995]: Waiting for error group managing segments... Dec 24 04:53:47.975268627 (445h24m47.846351662s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.976768513 (445h24m47.847851558s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.977666483 (445h24m47.848749528s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.979492480 (445h24m47.850575515s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.982702883 (445h24m47.853785919s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.990496404 (445h24m47.861579449s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.019123966 (445h24m47.890207001s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.205478756 (445h24m48.076561801s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.215568068 (445h24m48.086651114s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.255055000 (445h24m48.126138045s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.360364752 (445h24m48.231447787s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:14.990184584 (445h25m14.861267629s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:15.032082447 (445h25m14.903165492s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Dec 24 04:54:15.032375033 (445h25m14.903458078s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection Dec 24 04:54:15.032605533 (445h25m14.903688578s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, address:145.79.227.50:4375): protocol: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection Dec 24 04:54:15.032711815 (445h25m14.90379486s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection Dec 24 04:54:15.032736873 (445h25m14.903819908s) [seg_pos={0 0}, upload=390995]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:54:15.278462246 (445h25m15.149545291s) [upload=390995]: Done uploading segments... Dec 24 04:54:15.278513083 (445h25m15.149596138s) [upload=390995]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.637241403s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X..............X...] duration: 9.831968489s trace: https://jaeger.dev.storj.io/trace/1deda862aa4073e2 FAIL (2025-12-24 04:56:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:53:45.720403184 (445h24m45.591486219s) [upload=390995]: Starting upload Dec 24 04:53:45.720540295 (445h24m45.59162335s) [upload=390995]: Done starting upload Dec 24 04:53:45.720783568 (445h24m45.591866603s) [upload=390995]: Uploading segments... Dec 24 04:53:45.721560018 (445h24m45.592643083s) [upload=390995]: Got next segment. Inline: false Dec 24 04:53:45.721648986 (445h24m45.592732032s) [upload=390995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:53:45.721656201 (445h24m45.592739246s) [upload=390995]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:53:45.968344117 (445h24m45.839427152s) [seg_pos={0 0}, upload=390995]: Begin upload segment... Dec 24 04:53:45.968399892 (445h24m45.839482927s) [seg_pos={0 0}, upload=390995]: Done begin upload segment. Dec 24 04:53:45.968504732 (445h24m45.839587767s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.968571449 (445h24m45.839654484s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969513152 (445h24m45.840596187s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969756015 (445h24m45.84083905s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970245246 (445h24m45.841328281s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970597858 (445h24m45.841680903s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970970517 (445h24m45.842053553s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971362764 (445h24m45.842445799s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971712941 (445h24m45.842795986s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972101662 (445h24m45.843184697s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972560385 (445h24m45.84364342s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.974902435 (445h24m45.84598547s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.975744338 (445h24m45.846827374s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.976475501 (445h24m45.847558546s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.977774053 (445h24m45.848857098s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.978307899 (445h24m45.849390934s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:47.684950330 (445h24m47.556033365s) [upload=390995]: Next returned nil segment Dec 24 04:53:47.684967432 (445h24m47.556050477s) [upload=390995]: Waiting for error group managing segments... Dec 24 04:53:47.975268627 (445h24m47.846351662s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.976768513 (445h24m47.847851558s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.977666483 (445h24m47.848749528s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.979492480 (445h24m47.850575515s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.982702883 (445h24m47.853785919s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.990496404 (445h24m47.861579449s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.019123966 (445h24m47.890207001s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.205478756 (445h24m48.076561801s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.215568068 (445h24m48.086651114s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.255055000 (445h24m48.126138045s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.360364752 (445h24m48.231447787s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:14.990184584 (445h25m14.861267629s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:15.032082447 (445h25m14.903165492s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Dec 24 04:54:15.032375033 (445h25m14.903458078s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection Dec 24 04:54:15.032605533 (445h25m14.903688578s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, address:145.79.227.50:4375): protocol: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection Dec 24 04:54:15.032711815 (445h25m14.90379486s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection Dec 24 04:54:15.032736873 (445h25m14.903819908s) [seg_pos={0 0}, upload=390995]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:54:15.278462246 (445h25m15.149545291s) [upload=390995]: Done uploading segments... Dec 24 04:54:15.278513083 (445h25m15.149596138s) [upload=390995]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 9.831985161s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X..............X..] duration: 6.021244769s trace: https://jaeger.dev.storj.io/trace/1deda862aa4073e2 FAIL (2025-12-24 04:55:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:53:45.720403184 (445h24m45.591486219s) [upload=390995]: Starting upload Dec 24 04:53:45.720540295 (445h24m45.59162335s) [upload=390995]: Done starting upload Dec 24 04:53:45.720783568 (445h24m45.591866603s) [upload=390995]: Uploading segments... Dec 24 04:53:45.721560018 (445h24m45.592643083s) [upload=390995]: Got next segment. Inline: false Dec 24 04:53:45.721648986 (445h24m45.592732032s) [upload=390995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:53:45.721656201 (445h24m45.592739246s) [upload=390995]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:53:45.968344117 (445h24m45.839427152s) [seg_pos={0 0}, upload=390995]: Begin upload segment... Dec 24 04:53:45.968399892 (445h24m45.839482927s) [seg_pos={0 0}, upload=390995]: Done begin upload segment. Dec 24 04:53:45.968504732 (445h24m45.839587767s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.968571449 (445h24m45.839654484s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969513152 (445h24m45.840596187s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969756015 (445h24m45.84083905s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970245246 (445h24m45.841328281s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970597858 (445h24m45.841680903s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970970517 (445h24m45.842053553s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971362764 (445h24m45.842445799s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971712941 (445h24m45.842795986s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972101662 (445h24m45.843184697s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972560385 (445h24m45.84364342s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.974902435 (445h24m45.84598547s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.975744338 (445h24m45.846827374s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.976475501 (445h24m45.847558546s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.977774053 (445h24m45.848857098s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.978307899 (445h24m45.849390934s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:47.684950330 (445h24m47.556033365s) [upload=390995]: Next returned nil segment Dec 24 04:53:47.684967432 (445h24m47.556050477s) [upload=390995]: Waiting for error group managing segments... Dec 24 04:53:47.975268627 (445h24m47.846351662s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.976768513 (445h24m47.847851558s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.977666483 (445h24m47.848749528s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.979492480 (445h24m47.850575515s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.982702883 (445h24m47.853785919s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.990496404 (445h24m47.861579449s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.019123966 (445h24m47.890207001s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.205478756 (445h24m48.076561801s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.215568068 (445h24m48.086651114s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.255055000 (445h24m48.126138045s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.360364752 (445h24m48.231447787s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:14.990184584 (445h25m14.861267629s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:15.032082447 (445h25m14.903165492s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Dec 24 04:54:15.032375033 (445h25m14.903458078s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection Dec 24 04:54:15.032605533 (445h25m14.903688578s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, address:145.79.227.50:4375): protocol: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection Dec 24 04:54:15.032711815 (445h25m14.90379486s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection Dec 24 04:54:15.032736873 (445h25m14.903819908s) [seg_pos={0 0}, upload=390995]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:54:15.278462246 (445h25m15.149545291s) [upload=390995]: Done uploading segments... Dec 24 04:54:15.278513083 (445h25m15.149596138s) [upload=390995]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.021260839s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X..............X.] duration: 5.721355488s trace: https://jaeger.dev.storj.io/trace/1deda862aa4073e2 FAIL (2025-12-24 04:55:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:53:45.720403184 (445h24m45.591486219s) [upload=390995]: Starting upload Dec 24 04:53:45.720540295 (445h24m45.59162335s) [upload=390995]: Done starting upload Dec 24 04:53:45.720783568 (445h24m45.591866603s) [upload=390995]: Uploading segments... Dec 24 04:53:45.721560018 (445h24m45.592643083s) [upload=390995]: Got next segment. Inline: false Dec 24 04:53:45.721648986 (445h24m45.592732032s) [upload=390995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:53:45.721656201 (445h24m45.592739246s) [upload=390995]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:53:45.968344117 (445h24m45.839427152s) [seg_pos={0 0}, upload=390995]: Begin upload segment... Dec 24 04:53:45.968399892 (445h24m45.839482927s) [seg_pos={0 0}, upload=390995]: Done begin upload segment. Dec 24 04:53:45.968504732 (445h24m45.839587767s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.968571449 (445h24m45.839654484s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969513152 (445h24m45.840596187s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969756015 (445h24m45.84083905s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970245246 (445h24m45.841328281s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970597858 (445h24m45.841680903s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970970517 (445h24m45.842053553s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971362764 (445h24m45.842445799s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971712941 (445h24m45.842795986s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972101662 (445h24m45.843184697s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972560385 (445h24m45.84364342s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.974902435 (445h24m45.84598547s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.975744338 (445h24m45.846827374s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.976475501 (445h24m45.847558546s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.977774053 (445h24m45.848857098s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.978307899 (445h24m45.849390934s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:47.684950330 (445h24m47.556033365s) [upload=390995]: Next returned nil segment Dec 24 04:53:47.684967432 (445h24m47.556050477s) [upload=390995]: Waiting for error group managing segments... Dec 24 04:53:47.975268627 (445h24m47.846351662s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.976768513 (445h24m47.847851558s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.977666483 (445h24m47.848749528s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.979492480 (445h24m47.850575515s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.982702883 (445h24m47.853785919s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.990496404 (445h24m47.861579449s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.019123966 (445h24m47.890207001s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.205478756 (445h24m48.076561801s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.215568068 (445h24m48.086651114s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.255055000 (445h24m48.126138045s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.360364752 (445h24m48.231447787s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:14.990184584 (445h25m14.861267629s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:15.032082447 (445h25m14.903165492s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Dec 24 04:54:15.032375033 (445h25m14.903458078s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection Dec 24 04:54:15.032605533 (445h25m14.903688578s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, address:145.79.227.50:4375): protocol: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection Dec 24 04:54:15.032711815 (445h25m14.90379486s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection Dec 24 04:54:15.032736873 (445h25m14.903819908s) [seg_pos={0 0}, upload=390995]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:54:15.278462246 (445h25m15.149545291s) [upload=390995]: Done uploading segments... Dec 24 04:54:15.278513083 (445h25m15.149596138s) [upload=390995]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.721394461s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................X..............X] duration: 30.248701441s trace: https://jaeger.dev.storj.io/trace/1deda862aa4073e2 FAIL (2025-12-24 04:54:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:53:45.720403184 (445h24m45.591486219s) [upload=390995]: Starting upload Dec 24 04:53:45.720540295 (445h24m45.59162335s) [upload=390995]: Done starting upload Dec 24 04:53:45.720783568 (445h24m45.591866603s) [upload=390995]: Uploading segments... Dec 24 04:53:45.721560018 (445h24m45.592643083s) [upload=390995]: Got next segment. Inline: false Dec 24 04:53:45.721648986 (445h24m45.592732032s) [upload=390995]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:53:45.721656201 (445h24m45.592739246s) [upload=390995]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:53:45.968344117 (445h24m45.839427152s) [seg_pos={0 0}, upload=390995]: Begin upload segment... Dec 24 04:53:45.968399892 (445h24m45.839482927s) [seg_pos={0 0}, upload=390995]: Done begin upload segment. Dec 24 04:53:45.968504732 (445h24m45.839587767s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.968571449 (445h24m45.839654484s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969513152 (445h24m45.840596187s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.969756015 (445h24m45.84083905s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970245246 (445h24m45.841328281s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970597858 (445h24m45.841680903s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.970970517 (445h24m45.842053553s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971362764 (445h24m45.842445799s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.971712941 (445h24m45.842795986s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972101662 (445h24m45.843184697s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.972560385 (445h24m45.84364342s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.974902435 (445h24m45.84598547s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.975744338 (445h24m45.846827374s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.976475501 (445h24m45.847558546s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.977774053 (445h24m45.848857098s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:45.978307899 (445h24m45.849390934s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Uploading piece... Dec 24 04:53:47.684950330 (445h24m47.556033365s) [upload=390995]: Next returned nil segment Dec 24 04:53:47.684967432 (445h24m47.556050477s) [upload=390995]: Waiting for error group managing segments... Dec 24 04:53:47.975268627 (445h24m47.846351662s) [noise=true , address=145.79.227.49:4338 , piece_id=36TUWFBQRTGFTYP4CKI3TWVSRNTAXRYJMSNLRY27IFIWUPZHPXFQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.976768513 (445h24m47.847851558s) [noise=true , address=145.79.227.51:4411 , piece_id=3NL6L6GFNYFKKN7XSX7HCJH55PFYYSZ7S7TA4LXULPSFOY4WTCYQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.977666483 (445h24m47.848749528s) [noise=true , address=145.79.227.36:4518 , piece_id=UQ5SERQXXGKADBBNBREJDXO4S75EB52URJYV3GJQZR2IOYNGUPWQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.979492480 (445h24m47.850575515s) [noise=true , address=145.79.227.11:4729 , piece_id=KS5C52GNKPY7HOOSWUISW3VJU2N42X5DJBQVSHPN5MCQJNV5NGJA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.982702883 (445h24m47.853785919s) [noise=true , address=145.79.227.35:4483 , piece_id=72EIFGJ57XPERIIC5T2U46NPN2HQ2BNSESXR47BZCVZCOLIFSEDQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:47.990496404 (445h24m47.861579449s) [noise=true , address=145.79.227.34:4907 , piece_id=QR665WBFTK66K5MPE74VNJZMLAUQ72BR43E4TYZPT2NBHXUXI66A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.019123966 (445h24m47.890207001s) [noise=true , address=145.79.227.52:4445 , piece_id=32VTBLDKTMQ77JOJCWZLACZYOKJAQVMDBHX4ASEPBW6AA23HJKLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.205478756 (445h24m48.076561801s) [noise=true , address=45.91.116.8:4048 , piece_id=T5LFWAWQQDJLVXE3OVCA3VCISN442VK4KZRGOQFICZSGBVGPGC6Q, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.215568068 (445h24m48.086651114s) [noise=true , address=77.90.31.21:4880 , piece_id=YPSVIKZ3U6E2TMETASMUFGP2RH3FMYOTR4QC5GVB64R7JHVSGUHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.255055000 (445h24m48.126138045s) [noise=true , address=45.91.116.9:4262 , piece_id=3F4JFFTU2VIZBCHPMXE4W7NQEBEZLCIKSL2HXGMUCGNX2HFSU7XQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:53:48.360364752 (445h24m48.231447787s) [noise=true , address=45.91.116.14:4190 , piece_id=BAH772YJV2YX7TWOETNJXS6QMRE4ZP2SJ55TDJDCDEKO2T5L2PIA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:14.990184584 (445h25m14.861267629s) [noise=true , address=145.79.227.29:4942 , piece_id=SSYSLVPIIMNDFIC2WPLJQHOTZKEFYV23AADHCGS7WTCJIQNREVMQ, seg_pos={0 0}, upload=390995]: Done uploading piece. err: <nil> Dec 24 04:54:15.032082447 (445h25m14.903165492s) [noise=true , address=145.79.227.5:4223 , piece_id=WCCM2AN3RS6AU4EPQPRSH7RLIA2WC45TS53GLFDFG3XQ54MVAI7A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:145.79.227.5:4223): protocol: expected piece hash; context deadline exceeded; EOF Dec 24 04:54:15.032375033 (445h25m14.903458078s) [noise=true , address=145.79.227.32:4836 , piece_id=THEVAOMHWDFJIIHRN2OJHZGTOKG756JWBWHEEQW2TQYO5QC2RSLA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1jic2LawZWwB7J3oFbhRSM2YMurEqQQJM8UEEwWj25h3PxMugR, address:145.79.227.32:4836): protocol: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41810->145.79.227.32:4836: use of closed network connection Dec 24 04:54:15.032605533 (445h25m14.903688578s) [noise=true , address=145.79.227.50:4375 , piece_id=IK3JL6NFGO22YAMEXLFRXTQ5QZ7W7D2TK5RDSAGYBJ2HONRXCPHA, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5, address:145.79.227.50:4375): protocol: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48198->145.79.227.50:4375: use of closed network connection Dec 24 04:54:15.032711815 (445h25m14.90379486s) [noise=true , address=145.79.227.42:4121 , piece_id=YVEV2WVRQY4M3MITDBJOO7SVP45EHG5ZGCAXPAIMSHHGEGLVTS5A, seg_pos={0 0}, upload=390995]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47332->145.79.227.42:4121: use of closed network connection Dec 24 04:54:15.032736873 (445h25m14.903819908s) [seg_pos={0 0}, upload=390995]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:54:15.278462246 (445h25m15.149545291s) [upload=390995]: Done uploading segments... Dec 24 04:54:15.278513083 (445h25m15.149596138s) [upload=390995]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.248718814s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.824678846s trace: https://jaeger.dev.storj.io/trace/64b51f8bd213f167 FAIL (2025-12-24 04:52:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:42:53.854141273 (445h13m53.725224318s) [upload=390839]: Starting upload Dec 24 04:42:53.854215494 (445h13m53.725298539s) [upload=390839]: Done starting upload Dec 24 04:42:53.854479075 (445h13m53.72556212s) [upload=390839]: Uploading segments... Dec 24 04:42:53.854577853 (445h13m53.725660908s) [upload=390839]: Got next segment. Inline: false Dec 24 04:42:53.854618741 (445h13m53.725701776s) [upload=390839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:42:53.854626226 (445h13m53.725709261s) [upload=390839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:42:54.098954572 (445h13m53.970037617s) [seg_pos={0 0}, upload=390839]: Begin upload segment... Dec 24 04:42:54.099000660 (445h13m53.970083695s) [seg_pos={0 0}, upload=390839]: Done begin upload segment. Dec 24 04:42:54.099029364 (445h13m53.970112409s) [noise=true , address=145.79.227.39:4013 , piece_id=7KCQCZV7IS532ADB4DLESWHRUSB5ZWIZ25HW5YNWSFXWQB4TROSQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.099140977 (445h13m53.970224022s) [noise=true , address=77.90.31.14:4661 , piece_id=TODCW4FPSPAGVDWRIASF3PKOLXUNHPWSC62QPAWKSGYFNH4UKDEQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.099586515 (445h13m53.97066955s) [noise=true , address=145.79.227.46:4602 , piece_id=XNUM44DBNEMIIEYQ66MMOZSHIWN2O5QYV4LHK2452BP5O6EOST4A, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.099863232 (445h13m53.970946267s) [noise=true , address=145.79.227.50:4373 , piece_id=EDQNNC2DIWQTGGEA7KT6BPDJGIAHT42IQZPZ7YVC3FDHTASZ3WQQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.100147754 (445h13m53.971230799s) [noise=true , address=45.91.116.4:4939 , piece_id=DRA22PE2YDC3RYWT65F2POHKV47HEV22IGSYC5IPDVX2PQ7WDP2A, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.100455209 (445h13m53.971538244s) [noise=true , address=45.91.116.7:4010 , piece_id=KNG2IB4WHOMFXZM42EI25ML3EX6IPNWSXMDYTJUZMQYHCAW7YYWA, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.100748137 (445h13m53.971831172s) [noise=true , address=145.79.227.51:4410 , piece_id=2CEZTRDLXVAXFFEAEOF5SVHSOXRVB5ZTSQXYAN5GD4Y6ZU3DGV4Q, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.101029513 (445h13m53.972112548s) [noise=true , address=45.91.116.6:4974 , piece_id=DDIMQY46RCO5U3G3ZKWIIXAV44TUGRK5YTNVAF5OQPAKM5TJ637Q, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.101365643 (445h13m53.972448678s) [noise=true , address=145.79.227.40:4049 , piece_id=FODTLUWOIDSDZA5QFJ5RZPSYHRMPFOLK5NJCRUZ7GIXC236IRDQQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.101958713 (445h13m53.973041748s) [noise=true , address=77.90.31.19:4841 , piece_id=EUONIBKTAQGXYM6CAQVZ3FJWHJQAOVFBOCCF4XYLTQI4IU566XTQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.102487159 (445h13m53.973570204s) [noise=true , address=145.79.227.32:4834 , piece_id=CIY75P6ZSW5VKRMENI7AQA3PSYBACHVKGB4XAFQYKTDBZHT7AXRQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.102824290 (445h13m53.973907325s) [noise=true , address=77.90.31.12:4590 , piece_id=J74VB7H77KUHJQGQLD2SDLRPYVPPKIFFQL6ZZTNYZCBTP7ZNGXAQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.103165290 (445h13m53.974248325s) [noise=true , address=77.90.31.16:4736 , piece_id=E4BFE7Q4WTUPROYTG7IR4QLM4LL5XKCDS442KO654PB6BLEACSFQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.104097395 (445h13m53.97518043s) [noise=true , address=145.79.227.13:4804 , piece_id=L4LQOMARXANONKYYKIUKFH3H25CJ6XFAQ2D5YVILKHXHHFKGHP7A, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.106264812 (445h13m53.977347847s) [noise=true , address=145.79.227.24:4581 , piece_id=RKCKZQ7IXVKJ6WMQ7P2AQWTW5GYORBLFXAMTS7YBVY6YFK7ERCRA, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.107835482 (445h13m53.978918517s) [noise=true , address=145.79.227.49:4339 , piece_id=OE3ELR2OHWI27NPIY6C6HI2C6DK6LIVMCBFUTJK56TCFB4ABCXQQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:55.256583228 (445h13m55.127666264s) [noise=true , address=77.90.31.19:4841 , piece_id=EUONIBKTAQGXYM6CAQVZ3FJWHJQAOVFBOCCF4XYLTQI4IU566XTQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 24 04:42:55.825208676 (445h13m55.696291711s) [upload=390839]: Next returned nil segment Dec 24 04:42:55.825231309 (445h13m55.696314354s) [upload=390839]: Waiting for error group managing segments... Dec 24 04:42:56.111380959 (445h13m55.982463994s) [noise=true , address=145.79.227.46:4602 , piece_id=XNUM44DBNEMIIEYQ66MMOZSHIWN2O5QYV4LHK2452BP5O6EOST4A, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.112285412 (445h13m55.983368457s) [noise=true , address=145.79.227.49:4339 , piece_id=OE3ELR2OHWI27NPIY6C6HI2C6DK6LIVMCBFUTJK56TCFB4ABCXQQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.112648534 (445h13m55.983731579s) [noise=true , address=145.79.227.13:4804 , piece_id=L4LQOMARXANONKYYKIUKFH3H25CJ6XFAQ2D5YVILKHXHHFKGHP7A, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.129221522 (445h13m56.000304567s) [noise=true , address=145.79.227.50:4373 , piece_id=EDQNNC2DIWQTGGEA7KT6BPDJGIAHT42IQZPZ7YVC3FDHTASZ3WQQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.135066673 (445h13m56.006149708s) [noise=true , address=145.79.227.39:4013 , piece_id=7KCQCZV7IS532ADB4DLESWHRUSB5ZWIZ25HW5YNWSFXWQB4TROSQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.136122593 (445h13m56.007205628s) [noise=true , address=145.79.227.32:4834 , piece_id=CIY75P6ZSW5VKRMENI7AQA3PSYBACHVKGB4XAFQYKTDBZHT7AXRQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.297639110 (445h13m56.168722145s) [noise=true , address=77.90.31.16:4736 , piece_id=E4BFE7Q4WTUPROYTG7IR4QLM4LL5XKCDS442KO654PB6BLEACSFQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.327366527 (445h13m56.198449562s) [noise=true , address=45.91.116.4:4939 , piece_id=DRA22PE2YDC3RYWT65F2POHKV47HEV22IGSYC5IPDVX2PQ7WDP2A, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.353346166 (445h13m56.224429211s) [noise=true , address=77.90.31.14:4661 , piece_id=TODCW4FPSPAGVDWRIASF3PKOLXUNHPWSC62QPAWKSGYFNH4UKDEQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.362861025 (445h13m56.23394406s) [noise=true , address=45.91.116.7:4010 , piece_id=KNG2IB4WHOMFXZM42EI25ML3EX6IPNWSXMDYTJUZMQYHCAW7YYWA, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.373991459 (445h13m56.245074504s) [noise=true , address=77.90.31.12:4590 , piece_id=J74VB7H77KUHJQGQLD2SDLRPYVPPKIFFQL6ZZTNYZCBTP7ZNGXAQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.463096260 (445h13m56.334179305s) [noise=true , address=45.91.116.6:4974 , piece_id=DDIMQY46RCO5U3G3ZKWIIXAV44TUGRK5YTNVAF5OQPAKM5TJ637Q, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:43:23.169031756 (445h14m23.040114801s) [noise=true , address=145.79.227.40:4049 , piece_id=FODTLUWOIDSDZA5QFJ5RZPSYHRMPFOLK5NJCRUZ7GIXC236IRDQQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:39362->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:39362->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39362->145.79.227.40:4049: use of closed network connection Dec 24 04:43:23.169171592 (445h14m23.040254637s) [noise=true , address=145.79.227.51:4410 , piece_id=2CEZTRDLXVAXFFEAEOF5SVHSOXRVB5ZTSQXYAN5GD4Y6ZU3DGV4Q, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX, address:145.79.227.51:4410): protocol: write tcp 188.166.225.170:40626->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:40626->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40626->145.79.227.51:4410: use of closed network connection Dec 24 04:43:23.169262826 (445h14m23.040345872s) [noise=true , address=145.79.227.24:4581 , piece_id=RKCKZQ7IXVKJ6WMQ7P2AQWTW5GYORBLFXAMTS7YBVY6YFK7ERCRA, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:60164->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:60164->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60164->145.79.227.24:4581: use of closed network connection Dec 24 04:43:23.169311709 (445h14m23.040394754s) [seg_pos={0 0}, upload=390839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:43:23.405890860 (445h14m23.276973895s) [upload=390839]: Done uploading segments... Dec 24 04:43:23.405935244 (445h14m23.277018279s) [upload=390839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.82470714s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.735634958s trace: https://jaeger.dev.storj.io/trace/64b51f8bd213f167 FAIL (2025-12-24 04:52:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:42:53.854141273 (445h13m53.725224318s) [upload=390839]: Starting upload Dec 24 04:42:53.854215494 (445h13m53.725298539s) [upload=390839]: Done starting upload Dec 24 04:42:53.854479075 (445h13m53.72556212s) [upload=390839]: Uploading segments... Dec 24 04:42:53.854577853 (445h13m53.725660908s) [upload=390839]: Got next segment. Inline: false Dec 24 04:42:53.854618741 (445h13m53.725701776s) [upload=390839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:42:53.854626226 (445h13m53.725709261s) [upload=390839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:42:54.098954572 (445h13m53.970037617s) [seg_pos={0 0}, upload=390839]: Begin upload segment... Dec 24 04:42:54.099000660 (445h13m53.970083695s) [seg_pos={0 0}, upload=390839]: Done begin upload segment. Dec 24 04:42:54.099029364 (445h13m53.970112409s) [noise=true , address=145.79.227.39:4013 , piece_id=7KCQCZV7IS532ADB4DLESWHRUSB5ZWIZ25HW5YNWSFXWQB4TROSQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.099140977 (445h13m53.970224022s) [noise=true , address=77.90.31.14:4661 , piece_id=TODCW4FPSPAGVDWRIASF3PKOLXUNHPWSC62QPAWKSGYFNH4UKDEQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.099586515 (445h13m53.97066955s) [noise=true , address=145.79.227.46:4602 , piece_id=XNUM44DBNEMIIEYQ66MMOZSHIWN2O5QYV4LHK2452BP5O6EOST4A, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.099863232 (445h13m53.970946267s) [noise=true , address=145.79.227.50:4373 , piece_id=EDQNNC2DIWQTGGEA7KT6BPDJGIAHT42IQZPZ7YVC3FDHTASZ3WQQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.100147754 (445h13m53.971230799s) [noise=true , address=45.91.116.4:4939 , piece_id=DRA22PE2YDC3RYWT65F2POHKV47HEV22IGSYC5IPDVX2PQ7WDP2A, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.100455209 (445h13m53.971538244s) [noise=true , address=45.91.116.7:4010 , piece_id=KNG2IB4WHOMFXZM42EI25ML3EX6IPNWSXMDYTJUZMQYHCAW7YYWA, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.100748137 (445h13m53.971831172s) [noise=true , address=145.79.227.51:4410 , piece_id=2CEZTRDLXVAXFFEAEOF5SVHSOXRVB5ZTSQXYAN5GD4Y6ZU3DGV4Q, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.101029513 (445h13m53.972112548s) [noise=true , address=45.91.116.6:4974 , piece_id=DDIMQY46RCO5U3G3ZKWIIXAV44TUGRK5YTNVAF5OQPAKM5TJ637Q, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.101365643 (445h13m53.972448678s) [noise=true , address=145.79.227.40:4049 , piece_id=FODTLUWOIDSDZA5QFJ5RZPSYHRMPFOLK5NJCRUZ7GIXC236IRDQQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.101958713 (445h13m53.973041748s) [noise=true , address=77.90.31.19:4841 , piece_id=EUONIBKTAQGXYM6CAQVZ3FJWHJQAOVFBOCCF4XYLTQI4IU566XTQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.102487159 (445h13m53.973570204s) [noise=true , address=145.79.227.32:4834 , piece_id=CIY75P6ZSW5VKRMENI7AQA3PSYBACHVKGB4XAFQYKTDBZHT7AXRQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.102824290 (445h13m53.973907325s) [noise=true , address=77.90.31.12:4590 , piece_id=J74VB7H77KUHJQGQLD2SDLRPYVPPKIFFQL6ZZTNYZCBTP7ZNGXAQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.103165290 (445h13m53.974248325s) [noise=true , address=77.90.31.16:4736 , piece_id=E4BFE7Q4WTUPROYTG7IR4QLM4LL5XKCDS442KO654PB6BLEACSFQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.104097395 (445h13m53.97518043s) [noise=true , address=145.79.227.13:4804 , piece_id=L4LQOMARXANONKYYKIUKFH3H25CJ6XFAQ2D5YVILKHXHHFKGHP7A, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.106264812 (445h13m53.977347847s) [noise=true , address=145.79.227.24:4581 , piece_id=RKCKZQ7IXVKJ6WMQ7P2AQWTW5GYORBLFXAMTS7YBVY6YFK7ERCRA, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.107835482 (445h13m53.978918517s) [noise=true , address=145.79.227.49:4339 , piece_id=OE3ELR2OHWI27NPIY6C6HI2C6DK6LIVMCBFUTJK56TCFB4ABCXQQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:55.256583228 (445h13m55.127666264s) [noise=true , address=77.90.31.19:4841 , piece_id=EUONIBKTAQGXYM6CAQVZ3FJWHJQAOVFBOCCF4XYLTQI4IU566XTQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 24 04:42:55.825208676 (445h13m55.696291711s) [upload=390839]: Next returned nil segment Dec 24 04:42:55.825231309 (445h13m55.696314354s) [upload=390839]: Waiting for error group managing segments... Dec 24 04:42:56.111380959 (445h13m55.982463994s) [noise=true , address=145.79.227.46:4602 , piece_id=XNUM44DBNEMIIEYQ66MMOZSHIWN2O5QYV4LHK2452BP5O6EOST4A, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.112285412 (445h13m55.983368457s) [noise=true , address=145.79.227.49:4339 , piece_id=OE3ELR2OHWI27NPIY6C6HI2C6DK6LIVMCBFUTJK56TCFB4ABCXQQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.112648534 (445h13m55.983731579s) [noise=true , address=145.79.227.13:4804 , piece_id=L4LQOMARXANONKYYKIUKFH3H25CJ6XFAQ2D5YVILKHXHHFKGHP7A, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.129221522 (445h13m56.000304567s) [noise=true , address=145.79.227.50:4373 , piece_id=EDQNNC2DIWQTGGEA7KT6BPDJGIAHT42IQZPZ7YVC3FDHTASZ3WQQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.135066673 (445h13m56.006149708s) [noise=true , address=145.79.227.39:4013 , piece_id=7KCQCZV7IS532ADB4DLESWHRUSB5ZWIZ25HW5YNWSFXWQB4TROSQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.136122593 (445h13m56.007205628s) [noise=true , address=145.79.227.32:4834 , piece_id=CIY75P6ZSW5VKRMENI7AQA3PSYBACHVKGB4XAFQYKTDBZHT7AXRQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.297639110 (445h13m56.168722145s) [noise=true , address=77.90.31.16:4736 , piece_id=E4BFE7Q4WTUPROYTG7IR4QLM4LL5XKCDS442KO654PB6BLEACSFQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.327366527 (445h13m56.198449562s) [noise=true , address=45.91.116.4:4939 , piece_id=DRA22PE2YDC3RYWT65F2POHKV47HEV22IGSYC5IPDVX2PQ7WDP2A, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.353346166 (445h13m56.224429211s) [noise=true , address=77.90.31.14:4661 , piece_id=TODCW4FPSPAGVDWRIASF3PKOLXUNHPWSC62QPAWKSGYFNH4UKDEQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.362861025 (445h13m56.23394406s) [noise=true , address=45.91.116.7:4010 , piece_id=KNG2IB4WHOMFXZM42EI25ML3EX6IPNWSXMDYTJUZMQYHCAW7YYWA, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.373991459 (445h13m56.245074504s) [noise=true , address=77.90.31.12:4590 , piece_id=J74VB7H77KUHJQGQLD2SDLRPYVPPKIFFQL6ZZTNYZCBTP7ZNGXAQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.463096260 (445h13m56.334179305s) [noise=true , address=45.91.116.6:4974 , piece_id=DDIMQY46RCO5U3G3ZKWIIXAV44TUGRK5YTNVAF5OQPAKM5TJ637Q, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:43:23.169031756 (445h14m23.040114801s) [noise=true , address=145.79.227.40:4049 , piece_id=FODTLUWOIDSDZA5QFJ5RZPSYHRMPFOLK5NJCRUZ7GIXC236IRDQQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:39362->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:39362->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39362->145.79.227.40:4049: use of closed network connection Dec 24 04:43:23.169171592 (445h14m23.040254637s) [noise=true , address=145.79.227.51:4410 , piece_id=2CEZTRDLXVAXFFEAEOF5SVHSOXRVB5ZTSQXYAN5GD4Y6ZU3DGV4Q, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX, address:145.79.227.51:4410): protocol: write tcp 188.166.225.170:40626->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:40626->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40626->145.79.227.51:4410: use of closed network connection Dec 24 04:43:23.169262826 (445h14m23.040345872s) [noise=true , address=145.79.227.24:4581 , piece_id=RKCKZQ7IXVKJ6WMQ7P2AQWTW5GYORBLFXAMTS7YBVY6YFK7ERCRA, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:60164->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:60164->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60164->145.79.227.24:4581: use of closed network connection Dec 24 04:43:23.169311709 (445h14m23.040394754s) [seg_pos={0 0}, upload=390839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:43:23.405890860 (445h14m23.276973895s) [upload=390839]: Done uploading segments... Dec 24 04:43:23.405935244 (445h14m23.277018279s) [upload=390839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.735674343s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.695274377s trace: https://jaeger.dev.storj.io/trace/64b51f8bd213f167 FAIL (2025-12-24 04:51:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:42:53.854141273 (445h13m53.725224318s) [upload=390839]: Starting upload Dec 24 04:42:53.854215494 (445h13m53.725298539s) [upload=390839]: Done starting upload Dec 24 04:42:53.854479075 (445h13m53.72556212s) [upload=390839]: Uploading segments... Dec 24 04:42:53.854577853 (445h13m53.725660908s) [upload=390839]: Got next segment. Inline: false Dec 24 04:42:53.854618741 (445h13m53.725701776s) [upload=390839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:42:53.854626226 (445h13m53.725709261s) [upload=390839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:42:54.098954572 (445h13m53.970037617s) [seg_pos={0 0}, upload=390839]: Begin upload segment... Dec 24 04:42:54.099000660 (445h13m53.970083695s) [seg_pos={0 0}, upload=390839]: Done begin upload segment. Dec 24 04:42:54.099029364 (445h13m53.970112409s) [noise=true , address=145.79.227.39:4013 , piece_id=7KCQCZV7IS532ADB4DLESWHRUSB5ZWIZ25HW5YNWSFXWQB4TROSQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.099140977 (445h13m53.970224022s) [noise=true , address=77.90.31.14:4661 , piece_id=TODCW4FPSPAGVDWRIASF3PKOLXUNHPWSC62QPAWKSGYFNH4UKDEQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.099586515 (445h13m53.97066955s) [noise=true , address=145.79.227.46:4602 , piece_id=XNUM44DBNEMIIEYQ66MMOZSHIWN2O5QYV4LHK2452BP5O6EOST4A, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.099863232 (445h13m53.970946267s) [noise=true , address=145.79.227.50:4373 , piece_id=EDQNNC2DIWQTGGEA7KT6BPDJGIAHT42IQZPZ7YVC3FDHTASZ3WQQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.100147754 (445h13m53.971230799s) [noise=true , address=45.91.116.4:4939 , piece_id=DRA22PE2YDC3RYWT65F2POHKV47HEV22IGSYC5IPDVX2PQ7WDP2A, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.100455209 (445h13m53.971538244s) [noise=true , address=45.91.116.7:4010 , piece_id=KNG2IB4WHOMFXZM42EI25ML3EX6IPNWSXMDYTJUZMQYHCAW7YYWA, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.100748137 (445h13m53.971831172s) [noise=true , address=145.79.227.51:4410 , piece_id=2CEZTRDLXVAXFFEAEOF5SVHSOXRVB5ZTSQXYAN5GD4Y6ZU3DGV4Q, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.101029513 (445h13m53.972112548s) [noise=true , address=45.91.116.6:4974 , piece_id=DDIMQY46RCO5U3G3ZKWIIXAV44TUGRK5YTNVAF5OQPAKM5TJ637Q, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.101365643 (445h13m53.972448678s) [noise=true , address=145.79.227.40:4049 , piece_id=FODTLUWOIDSDZA5QFJ5RZPSYHRMPFOLK5NJCRUZ7GIXC236IRDQQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.101958713 (445h13m53.973041748s) [noise=true , address=77.90.31.19:4841 , piece_id=EUONIBKTAQGXYM6CAQVZ3FJWHJQAOVFBOCCF4XYLTQI4IU566XTQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.102487159 (445h13m53.973570204s) [noise=true , address=145.79.227.32:4834 , piece_id=CIY75P6ZSW5VKRMENI7AQA3PSYBACHVKGB4XAFQYKTDBZHT7AXRQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.102824290 (445h13m53.973907325s) [noise=true , address=77.90.31.12:4590 , piece_id=J74VB7H77KUHJQGQLD2SDLRPYVPPKIFFQL6ZZTNYZCBTP7ZNGXAQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.103165290 (445h13m53.974248325s) [noise=true , address=77.90.31.16:4736 , piece_id=E4BFE7Q4WTUPROYTG7IR4QLM4LL5XKCDS442KO654PB6BLEACSFQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.104097395 (445h13m53.97518043s) [noise=true , address=145.79.227.13:4804 , piece_id=L4LQOMARXANONKYYKIUKFH3H25CJ6XFAQ2D5YVILKHXHHFKGHP7A, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.106264812 (445h13m53.977347847s) [noise=true , address=145.79.227.24:4581 , piece_id=RKCKZQ7IXVKJ6WMQ7P2AQWTW5GYORBLFXAMTS7YBVY6YFK7ERCRA, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.107835482 (445h13m53.978918517s) [noise=true , address=145.79.227.49:4339 , piece_id=OE3ELR2OHWI27NPIY6C6HI2C6DK6LIVMCBFUTJK56TCFB4ABCXQQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:55.256583228 (445h13m55.127666264s) [noise=true , address=77.90.31.19:4841 , piece_id=EUONIBKTAQGXYM6CAQVZ3FJWHJQAOVFBOCCF4XYLTQI4IU566XTQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 24 04:42:55.825208676 (445h13m55.696291711s) [upload=390839]: Next returned nil segment Dec 24 04:42:55.825231309 (445h13m55.696314354s) [upload=390839]: Waiting for error group managing segments... Dec 24 04:42:56.111380959 (445h13m55.982463994s) [noise=true , address=145.79.227.46:4602 , piece_id=XNUM44DBNEMIIEYQ66MMOZSHIWN2O5QYV4LHK2452BP5O6EOST4A, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.112285412 (445h13m55.983368457s) [noise=true , address=145.79.227.49:4339 , piece_id=OE3ELR2OHWI27NPIY6C6HI2C6DK6LIVMCBFUTJK56TCFB4ABCXQQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.112648534 (445h13m55.983731579s) [noise=true , address=145.79.227.13:4804 , piece_id=L4LQOMARXANONKYYKIUKFH3H25CJ6XFAQ2D5YVILKHXHHFKGHP7A, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.129221522 (445h13m56.000304567s) [noise=true , address=145.79.227.50:4373 , piece_id=EDQNNC2DIWQTGGEA7KT6BPDJGIAHT42IQZPZ7YVC3FDHTASZ3WQQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.135066673 (445h13m56.006149708s) [noise=true , address=145.79.227.39:4013 , piece_id=7KCQCZV7IS532ADB4DLESWHRUSB5ZWIZ25HW5YNWSFXWQB4TROSQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.136122593 (445h13m56.007205628s) [noise=true , address=145.79.227.32:4834 , piece_id=CIY75P6ZSW5VKRMENI7AQA3PSYBACHVKGB4XAFQYKTDBZHT7AXRQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.297639110 (445h13m56.168722145s) [noise=true , address=77.90.31.16:4736 , piece_id=E4BFE7Q4WTUPROYTG7IR4QLM4LL5XKCDS442KO654PB6BLEACSFQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.327366527 (445h13m56.198449562s) [noise=true , address=45.91.116.4:4939 , piece_id=DRA22PE2YDC3RYWT65F2POHKV47HEV22IGSYC5IPDVX2PQ7WDP2A, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.353346166 (445h13m56.224429211s) [noise=true , address=77.90.31.14:4661 , piece_id=TODCW4FPSPAGVDWRIASF3PKOLXUNHPWSC62QPAWKSGYFNH4UKDEQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.362861025 (445h13m56.23394406s) [noise=true , address=45.91.116.7:4010 , piece_id=KNG2IB4WHOMFXZM42EI25ML3EX6IPNWSXMDYTJUZMQYHCAW7YYWA, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.373991459 (445h13m56.245074504s) [noise=true , address=77.90.31.12:4590 , piece_id=J74VB7H77KUHJQGQLD2SDLRPYVPPKIFFQL6ZZTNYZCBTP7ZNGXAQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.463096260 (445h13m56.334179305s) [noise=true , address=45.91.116.6:4974 , piece_id=DDIMQY46RCO5U3G3ZKWIIXAV44TUGRK5YTNVAF5OQPAKM5TJ637Q, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:43:23.169031756 (445h14m23.040114801s) [noise=true , address=145.79.227.40:4049 , piece_id=FODTLUWOIDSDZA5QFJ5RZPSYHRMPFOLK5NJCRUZ7GIXC236IRDQQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:39362->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:39362->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39362->145.79.227.40:4049: use of closed network connection Dec 24 04:43:23.169171592 (445h14m23.040254637s) [noise=true , address=145.79.227.51:4410 , piece_id=2CEZTRDLXVAXFFEAEOF5SVHSOXRVB5ZTSQXYAN5GD4Y6ZU3DGV4Q, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX, address:145.79.227.51:4410): protocol: write tcp 188.166.225.170:40626->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:40626->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40626->145.79.227.51:4410: use of closed network connection Dec 24 04:43:23.169262826 (445h14m23.040345872s) [noise=true , address=145.79.227.24:4581 , piece_id=RKCKZQ7IXVKJ6WMQ7P2AQWTW5GYORBLFXAMTS7YBVY6YFK7ERCRA, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:60164->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:60164->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60164->145.79.227.24:4581: use of closed network connection Dec 24 04:43:23.169311709 (445h14m23.040394754s) [seg_pos={0 0}, upload=390839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:43:23.405890860 (445h14m23.276973895s) [upload=390839]: Done uploading segments... Dec 24 04:43:23.405935244 (445h14m23.277018279s) [upload=390839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.695306888s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.727973644s trace: https://jaeger.dev.storj.io/trace/64b51f8bd213f167 FAIL (2025-12-24 04:50:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:42:53.854141273 (445h13m53.725224318s) [upload=390839]: Starting upload Dec 24 04:42:53.854215494 (445h13m53.725298539s) [upload=390839]: Done starting upload Dec 24 04:42:53.854479075 (445h13m53.72556212s) [upload=390839]: Uploading segments... Dec 24 04:42:53.854577853 (445h13m53.725660908s) [upload=390839]: Got next segment. Inline: false Dec 24 04:42:53.854618741 (445h13m53.725701776s) [upload=390839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:42:53.854626226 (445h13m53.725709261s) [upload=390839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:42:54.098954572 (445h13m53.970037617s) [seg_pos={0 0}, upload=390839]: Begin upload segment... Dec 24 04:42:54.099000660 (445h13m53.970083695s) [seg_pos={0 0}, upload=390839]: Done begin upload segment. Dec 24 04:42:54.099029364 (445h13m53.970112409s) [noise=true , address=145.79.227.39:4013 , piece_id=7KCQCZV7IS532ADB4DLESWHRUSB5ZWIZ25HW5YNWSFXWQB4TROSQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.099140977 (445h13m53.970224022s) [noise=true , address=77.90.31.14:4661 , piece_id=TODCW4FPSPAGVDWRIASF3PKOLXUNHPWSC62QPAWKSGYFNH4UKDEQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.099586515 (445h13m53.97066955s) [noise=true , address=145.79.227.46:4602 , piece_id=XNUM44DBNEMIIEYQ66MMOZSHIWN2O5QYV4LHK2452BP5O6EOST4A, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.099863232 (445h13m53.970946267s) [noise=true , address=145.79.227.50:4373 , piece_id=EDQNNC2DIWQTGGEA7KT6BPDJGIAHT42IQZPZ7YVC3FDHTASZ3WQQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.100147754 (445h13m53.971230799s) [noise=true , address=45.91.116.4:4939 , piece_id=DRA22PE2YDC3RYWT65F2POHKV47HEV22IGSYC5IPDVX2PQ7WDP2A, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.100455209 (445h13m53.971538244s) [noise=true , address=45.91.116.7:4010 , piece_id=KNG2IB4WHOMFXZM42EI25ML3EX6IPNWSXMDYTJUZMQYHCAW7YYWA, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.100748137 (445h13m53.971831172s) [noise=true , address=145.79.227.51:4410 , piece_id=2CEZTRDLXVAXFFEAEOF5SVHSOXRVB5ZTSQXYAN5GD4Y6ZU3DGV4Q, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.101029513 (445h13m53.972112548s) [noise=true , address=45.91.116.6:4974 , piece_id=DDIMQY46RCO5U3G3ZKWIIXAV44TUGRK5YTNVAF5OQPAKM5TJ637Q, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.101365643 (445h13m53.972448678s) [noise=true , address=145.79.227.40:4049 , piece_id=FODTLUWOIDSDZA5QFJ5RZPSYHRMPFOLK5NJCRUZ7GIXC236IRDQQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.101958713 (445h13m53.973041748s) [noise=true , address=77.90.31.19:4841 , piece_id=EUONIBKTAQGXYM6CAQVZ3FJWHJQAOVFBOCCF4XYLTQI4IU566XTQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.102487159 (445h13m53.973570204s) [noise=true , address=145.79.227.32:4834 , piece_id=CIY75P6ZSW5VKRMENI7AQA3PSYBACHVKGB4XAFQYKTDBZHT7AXRQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.102824290 (445h13m53.973907325s) [noise=true , address=77.90.31.12:4590 , piece_id=J74VB7H77KUHJQGQLD2SDLRPYVPPKIFFQL6ZZTNYZCBTP7ZNGXAQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.103165290 (445h13m53.974248325s) [noise=true , address=77.90.31.16:4736 , piece_id=E4BFE7Q4WTUPROYTG7IR4QLM4LL5XKCDS442KO654PB6BLEACSFQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.104097395 (445h13m53.97518043s) [noise=true , address=145.79.227.13:4804 , piece_id=L4LQOMARXANONKYYKIUKFH3H25CJ6XFAQ2D5YVILKHXHHFKGHP7A, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.106264812 (445h13m53.977347847s) [noise=true , address=145.79.227.24:4581 , piece_id=RKCKZQ7IXVKJ6WMQ7P2AQWTW5GYORBLFXAMTS7YBVY6YFK7ERCRA, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.107835482 (445h13m53.978918517s) [noise=true , address=145.79.227.49:4339 , piece_id=OE3ELR2OHWI27NPIY6C6HI2C6DK6LIVMCBFUTJK56TCFB4ABCXQQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:55.256583228 (445h13m55.127666264s) [noise=true , address=77.90.31.19:4841 , piece_id=EUONIBKTAQGXYM6CAQVZ3FJWHJQAOVFBOCCF4XYLTQI4IU566XTQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 24 04:42:55.825208676 (445h13m55.696291711s) [upload=390839]: Next returned nil segment Dec 24 04:42:55.825231309 (445h13m55.696314354s) [upload=390839]: Waiting for error group managing segments... Dec 24 04:42:56.111380959 (445h13m55.982463994s) [noise=true , address=145.79.227.46:4602 , piece_id=XNUM44DBNEMIIEYQ66MMOZSHIWN2O5QYV4LHK2452BP5O6EOST4A, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.112285412 (445h13m55.983368457s) [noise=true , address=145.79.227.49:4339 , piece_id=OE3ELR2OHWI27NPIY6C6HI2C6DK6LIVMCBFUTJK56TCFB4ABCXQQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.112648534 (445h13m55.983731579s) [noise=true , address=145.79.227.13:4804 , piece_id=L4LQOMARXANONKYYKIUKFH3H25CJ6XFAQ2D5YVILKHXHHFKGHP7A, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.129221522 (445h13m56.000304567s) [noise=true , address=145.79.227.50:4373 , piece_id=EDQNNC2DIWQTGGEA7KT6BPDJGIAHT42IQZPZ7YVC3FDHTASZ3WQQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.135066673 (445h13m56.006149708s) [noise=true , address=145.79.227.39:4013 , piece_id=7KCQCZV7IS532ADB4DLESWHRUSB5ZWIZ25HW5YNWSFXWQB4TROSQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.136122593 (445h13m56.007205628s) [noise=true , address=145.79.227.32:4834 , piece_id=CIY75P6ZSW5VKRMENI7AQA3PSYBACHVKGB4XAFQYKTDBZHT7AXRQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.297639110 (445h13m56.168722145s) [noise=true , address=77.90.31.16:4736 , piece_id=E4BFE7Q4WTUPROYTG7IR4QLM4LL5XKCDS442KO654PB6BLEACSFQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.327366527 (445h13m56.198449562s) [noise=true , address=45.91.116.4:4939 , piece_id=DRA22PE2YDC3RYWT65F2POHKV47HEV22IGSYC5IPDVX2PQ7WDP2A, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.353346166 (445h13m56.224429211s) [noise=true , address=77.90.31.14:4661 , piece_id=TODCW4FPSPAGVDWRIASF3PKOLXUNHPWSC62QPAWKSGYFNH4UKDEQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.362861025 (445h13m56.23394406s) [noise=true , address=45.91.116.7:4010 , piece_id=KNG2IB4WHOMFXZM42EI25ML3EX6IPNWSXMDYTJUZMQYHCAW7YYWA, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.373991459 (445h13m56.245074504s) [noise=true , address=77.90.31.12:4590 , piece_id=J74VB7H77KUHJQGQLD2SDLRPYVPPKIFFQL6ZZTNYZCBTP7ZNGXAQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.463096260 (445h13m56.334179305s) [noise=true , address=45.91.116.6:4974 , piece_id=DDIMQY46RCO5U3G3ZKWIIXAV44TUGRK5YTNVAF5OQPAKM5TJ637Q, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:43:23.169031756 (445h14m23.040114801s) [noise=true , address=145.79.227.40:4049 , piece_id=FODTLUWOIDSDZA5QFJ5RZPSYHRMPFOLK5NJCRUZ7GIXC236IRDQQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:39362->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:39362->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39362->145.79.227.40:4049: use of closed network connection Dec 24 04:43:23.169171592 (445h14m23.040254637s) [noise=true , address=145.79.227.51:4410 , piece_id=2CEZTRDLXVAXFFEAEOF5SVHSOXRVB5ZTSQXYAN5GD4Y6ZU3DGV4Q, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX, address:145.79.227.51:4410): protocol: write tcp 188.166.225.170:40626->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:40626->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40626->145.79.227.51:4410: use of closed network connection Dec 24 04:43:23.169262826 (445h14m23.040345872s) [noise=true , address=145.79.227.24:4581 , piece_id=RKCKZQ7IXVKJ6WMQ7P2AQWTW5GYORBLFXAMTS7YBVY6YFK7ERCRA, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:60164->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:60164->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60164->145.79.227.24:4581: use of closed network connection Dec 24 04:43:23.169311709 (445h14m23.040394754s) [seg_pos={0 0}, upload=390839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:43:23.405890860 (445h14m23.276973895s) [upload=390839]: Done uploading segments... Dec 24 04:43:23.405935244 (445h14m23.277018279s) [upload=390839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.728003981s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 20.581757674s trace: https://jaeger.dev.storj.io/trace/64b51f8bd213f167 FAIL (2025-12-24 04:50:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:42:53.854141273 (445h13m53.725224318s) [upload=390839]: Starting upload Dec 24 04:42:53.854215494 (445h13m53.725298539s) [upload=390839]: Done starting upload Dec 24 04:42:53.854479075 (445h13m53.72556212s) [upload=390839]: Uploading segments... Dec 24 04:42:53.854577853 (445h13m53.725660908s) [upload=390839]: Got next segment. Inline: false Dec 24 04:42:53.854618741 (445h13m53.725701776s) [upload=390839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:42:53.854626226 (445h13m53.725709261s) [upload=390839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:42:54.098954572 (445h13m53.970037617s) [seg_pos={0 0}, upload=390839]: Begin upload segment... Dec 24 04:42:54.099000660 (445h13m53.970083695s) [seg_pos={0 0}, upload=390839]: Done begin upload segment. Dec 24 04:42:54.099029364 (445h13m53.970112409s) [noise=true , address=145.79.227.39:4013 , piece_id=7KCQCZV7IS532ADB4DLESWHRUSB5ZWIZ25HW5YNWSFXWQB4TROSQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.099140977 (445h13m53.970224022s) [noise=true , address=77.90.31.14:4661 , piece_id=TODCW4FPSPAGVDWRIASF3PKOLXUNHPWSC62QPAWKSGYFNH4UKDEQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.099586515 (445h13m53.97066955s) [noise=true , address=145.79.227.46:4602 , piece_id=XNUM44DBNEMIIEYQ66MMOZSHIWN2O5QYV4LHK2452BP5O6EOST4A, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.099863232 (445h13m53.970946267s) [noise=true , address=145.79.227.50:4373 , piece_id=EDQNNC2DIWQTGGEA7KT6BPDJGIAHT42IQZPZ7YVC3FDHTASZ3WQQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.100147754 (445h13m53.971230799s) [noise=true , address=45.91.116.4:4939 , piece_id=DRA22PE2YDC3RYWT65F2POHKV47HEV22IGSYC5IPDVX2PQ7WDP2A, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.100455209 (445h13m53.971538244s) [noise=true , address=45.91.116.7:4010 , piece_id=KNG2IB4WHOMFXZM42EI25ML3EX6IPNWSXMDYTJUZMQYHCAW7YYWA, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.100748137 (445h13m53.971831172s) [noise=true , address=145.79.227.51:4410 , piece_id=2CEZTRDLXVAXFFEAEOF5SVHSOXRVB5ZTSQXYAN5GD4Y6ZU3DGV4Q, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.101029513 (445h13m53.972112548s) [noise=true , address=45.91.116.6:4974 , piece_id=DDIMQY46RCO5U3G3ZKWIIXAV44TUGRK5YTNVAF5OQPAKM5TJ637Q, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.101365643 (445h13m53.972448678s) [noise=true , address=145.79.227.40:4049 , piece_id=FODTLUWOIDSDZA5QFJ5RZPSYHRMPFOLK5NJCRUZ7GIXC236IRDQQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.101958713 (445h13m53.973041748s) [noise=true , address=77.90.31.19:4841 , piece_id=EUONIBKTAQGXYM6CAQVZ3FJWHJQAOVFBOCCF4XYLTQI4IU566XTQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.102487159 (445h13m53.973570204s) [noise=true , address=145.79.227.32:4834 , piece_id=CIY75P6ZSW5VKRMENI7AQA3PSYBACHVKGB4XAFQYKTDBZHT7AXRQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.102824290 (445h13m53.973907325s) [noise=true , address=77.90.31.12:4590 , piece_id=J74VB7H77KUHJQGQLD2SDLRPYVPPKIFFQL6ZZTNYZCBTP7ZNGXAQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.103165290 (445h13m53.974248325s) [noise=true , address=77.90.31.16:4736 , piece_id=E4BFE7Q4WTUPROYTG7IR4QLM4LL5XKCDS442KO654PB6BLEACSFQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.104097395 (445h13m53.97518043s) [noise=true , address=145.79.227.13:4804 , piece_id=L4LQOMARXANONKYYKIUKFH3H25CJ6XFAQ2D5YVILKHXHHFKGHP7A, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.106264812 (445h13m53.977347847s) [noise=true , address=145.79.227.24:4581 , piece_id=RKCKZQ7IXVKJ6WMQ7P2AQWTW5GYORBLFXAMTS7YBVY6YFK7ERCRA, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.107835482 (445h13m53.978918517s) [noise=true , address=145.79.227.49:4339 , piece_id=OE3ELR2OHWI27NPIY6C6HI2C6DK6LIVMCBFUTJK56TCFB4ABCXQQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:55.256583228 (445h13m55.127666264s) [noise=true , address=77.90.31.19:4841 , piece_id=EUONIBKTAQGXYM6CAQVZ3FJWHJQAOVFBOCCF4XYLTQI4IU566XTQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 24 04:42:55.825208676 (445h13m55.696291711s) [upload=390839]: Next returned nil segment Dec 24 04:42:55.825231309 (445h13m55.696314354s) [upload=390839]: Waiting for error group managing segments... Dec 24 04:42:56.111380959 (445h13m55.982463994s) [noise=true , address=145.79.227.46:4602 , piece_id=XNUM44DBNEMIIEYQ66MMOZSHIWN2O5QYV4LHK2452BP5O6EOST4A, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.112285412 (445h13m55.983368457s) [noise=true , address=145.79.227.49:4339 , piece_id=OE3ELR2OHWI27NPIY6C6HI2C6DK6LIVMCBFUTJK56TCFB4ABCXQQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.112648534 (445h13m55.983731579s) [noise=true , address=145.79.227.13:4804 , piece_id=L4LQOMARXANONKYYKIUKFH3H25CJ6XFAQ2D5YVILKHXHHFKGHP7A, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.129221522 (445h13m56.000304567s) [noise=true , address=145.79.227.50:4373 , piece_id=EDQNNC2DIWQTGGEA7KT6BPDJGIAHT42IQZPZ7YVC3FDHTASZ3WQQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.135066673 (445h13m56.006149708s) [noise=true , address=145.79.227.39:4013 , piece_id=7KCQCZV7IS532ADB4DLESWHRUSB5ZWIZ25HW5YNWSFXWQB4TROSQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.136122593 (445h13m56.007205628s) [noise=true , address=145.79.227.32:4834 , piece_id=CIY75P6ZSW5VKRMENI7AQA3PSYBACHVKGB4XAFQYKTDBZHT7AXRQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.297639110 (445h13m56.168722145s) [noise=true , address=77.90.31.16:4736 , piece_id=E4BFE7Q4WTUPROYTG7IR4QLM4LL5XKCDS442KO654PB6BLEACSFQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.327366527 (445h13m56.198449562s) [noise=true , address=45.91.116.4:4939 , piece_id=DRA22PE2YDC3RYWT65F2POHKV47HEV22IGSYC5IPDVX2PQ7WDP2A, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.353346166 (445h13m56.224429211s) [noise=true , address=77.90.31.14:4661 , piece_id=TODCW4FPSPAGVDWRIASF3PKOLXUNHPWSC62QPAWKSGYFNH4UKDEQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.362861025 (445h13m56.23394406s) [noise=true , address=45.91.116.7:4010 , piece_id=KNG2IB4WHOMFXZM42EI25ML3EX6IPNWSXMDYTJUZMQYHCAW7YYWA, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.373991459 (445h13m56.245074504s) [noise=true , address=77.90.31.12:4590 , piece_id=J74VB7H77KUHJQGQLD2SDLRPYVPPKIFFQL6ZZTNYZCBTP7ZNGXAQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.463096260 (445h13m56.334179305s) [noise=true , address=45.91.116.6:4974 , piece_id=DDIMQY46RCO5U3G3ZKWIIXAV44TUGRK5YTNVAF5OQPAKM5TJ637Q, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:43:23.169031756 (445h14m23.040114801s) [noise=true , address=145.79.227.40:4049 , piece_id=FODTLUWOIDSDZA5QFJ5RZPSYHRMPFOLK5NJCRUZ7GIXC236IRDQQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:39362->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:39362->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39362->145.79.227.40:4049: use of closed network connection Dec 24 04:43:23.169171592 (445h14m23.040254637s) [noise=true , address=145.79.227.51:4410 , piece_id=2CEZTRDLXVAXFFEAEOF5SVHSOXRVB5ZTSQXYAN5GD4Y6ZU3DGV4Q, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX, address:145.79.227.51:4410): protocol: write tcp 188.166.225.170:40626->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:40626->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40626->145.79.227.51:4410: use of closed network connection Dec 24 04:43:23.169262826 (445h14m23.040345872s) [noise=true , address=145.79.227.24:4581 , piece_id=RKCKZQ7IXVKJ6WMQ7P2AQWTW5GYORBLFXAMTS7YBVY6YFK7ERCRA, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:60164->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:60164->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60164->145.79.227.24:4581: use of closed network connection Dec 24 04:43:23.169311709 (445h14m23.040394754s) [seg_pos={0 0}, upload=390839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:43:23.405890860 (445h14m23.276973895s) [upload=390839]: Done uploading segments... Dec 24 04:43:23.405935244 (445h14m23.277018279s) [upload=390839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 20.581796698s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.896366765s trace: https://jaeger.dev.storj.io/trace/64b51f8bd213f167 FAIL (2025-12-24 04:49:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:42:53.854141273 (445h13m53.725224318s) [upload=390839]: Starting upload Dec 24 04:42:53.854215494 (445h13m53.725298539s) [upload=390839]: Done starting upload Dec 24 04:42:53.854479075 (445h13m53.72556212s) [upload=390839]: Uploading segments... Dec 24 04:42:53.854577853 (445h13m53.725660908s) [upload=390839]: Got next segment. Inline: false Dec 24 04:42:53.854618741 (445h13m53.725701776s) [upload=390839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:42:53.854626226 (445h13m53.725709261s) [upload=390839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:42:54.098954572 (445h13m53.970037617s) [seg_pos={0 0}, upload=390839]: Begin upload segment... Dec 24 04:42:54.099000660 (445h13m53.970083695s) [seg_pos={0 0}, upload=390839]: Done begin upload segment. Dec 24 04:42:54.099029364 (445h13m53.970112409s) [noise=true , address=145.79.227.39:4013 , piece_id=7KCQCZV7IS532ADB4DLESWHRUSB5ZWIZ25HW5YNWSFXWQB4TROSQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.099140977 (445h13m53.970224022s) [noise=true , address=77.90.31.14:4661 , piece_id=TODCW4FPSPAGVDWRIASF3PKOLXUNHPWSC62QPAWKSGYFNH4UKDEQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.099586515 (445h13m53.97066955s) [noise=true , address=145.79.227.46:4602 , piece_id=XNUM44DBNEMIIEYQ66MMOZSHIWN2O5QYV4LHK2452BP5O6EOST4A, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.099863232 (445h13m53.970946267s) [noise=true , address=145.79.227.50:4373 , piece_id=EDQNNC2DIWQTGGEA7KT6BPDJGIAHT42IQZPZ7YVC3FDHTASZ3WQQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.100147754 (445h13m53.971230799s) [noise=true , address=45.91.116.4:4939 , piece_id=DRA22PE2YDC3RYWT65F2POHKV47HEV22IGSYC5IPDVX2PQ7WDP2A, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.100455209 (445h13m53.971538244s) [noise=true , address=45.91.116.7:4010 , piece_id=KNG2IB4WHOMFXZM42EI25ML3EX6IPNWSXMDYTJUZMQYHCAW7YYWA, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.100748137 (445h13m53.971831172s) [noise=true , address=145.79.227.51:4410 , piece_id=2CEZTRDLXVAXFFEAEOF5SVHSOXRVB5ZTSQXYAN5GD4Y6ZU3DGV4Q, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.101029513 (445h13m53.972112548s) [noise=true , address=45.91.116.6:4974 , piece_id=DDIMQY46RCO5U3G3ZKWIIXAV44TUGRK5YTNVAF5OQPAKM5TJ637Q, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.101365643 (445h13m53.972448678s) [noise=true , address=145.79.227.40:4049 , piece_id=FODTLUWOIDSDZA5QFJ5RZPSYHRMPFOLK5NJCRUZ7GIXC236IRDQQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.101958713 (445h13m53.973041748s) [noise=true , address=77.90.31.19:4841 , piece_id=EUONIBKTAQGXYM6CAQVZ3FJWHJQAOVFBOCCF4XYLTQI4IU566XTQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.102487159 (445h13m53.973570204s) [noise=true , address=145.79.227.32:4834 , piece_id=CIY75P6ZSW5VKRMENI7AQA3PSYBACHVKGB4XAFQYKTDBZHT7AXRQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.102824290 (445h13m53.973907325s) [noise=true , address=77.90.31.12:4590 , piece_id=J74VB7H77KUHJQGQLD2SDLRPYVPPKIFFQL6ZZTNYZCBTP7ZNGXAQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.103165290 (445h13m53.974248325s) [noise=true , address=77.90.31.16:4736 , piece_id=E4BFE7Q4WTUPROYTG7IR4QLM4LL5XKCDS442KO654PB6BLEACSFQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.104097395 (445h13m53.97518043s) [noise=true , address=145.79.227.13:4804 , piece_id=L4LQOMARXANONKYYKIUKFH3H25CJ6XFAQ2D5YVILKHXHHFKGHP7A, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.106264812 (445h13m53.977347847s) [noise=true , address=145.79.227.24:4581 , piece_id=RKCKZQ7IXVKJ6WMQ7P2AQWTW5GYORBLFXAMTS7YBVY6YFK7ERCRA, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.107835482 (445h13m53.978918517s) [noise=true , address=145.79.227.49:4339 , piece_id=OE3ELR2OHWI27NPIY6C6HI2C6DK6LIVMCBFUTJK56TCFB4ABCXQQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:55.256583228 (445h13m55.127666264s) [noise=true , address=77.90.31.19:4841 , piece_id=EUONIBKTAQGXYM6CAQVZ3FJWHJQAOVFBOCCF4XYLTQI4IU566XTQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 24 04:42:55.825208676 (445h13m55.696291711s) [upload=390839]: Next returned nil segment Dec 24 04:42:55.825231309 (445h13m55.696314354s) [upload=390839]: Waiting for error group managing segments... Dec 24 04:42:56.111380959 (445h13m55.982463994s) [noise=true , address=145.79.227.46:4602 , piece_id=XNUM44DBNEMIIEYQ66MMOZSHIWN2O5QYV4LHK2452BP5O6EOST4A, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.112285412 (445h13m55.983368457s) [noise=true , address=145.79.227.49:4339 , piece_id=OE3ELR2OHWI27NPIY6C6HI2C6DK6LIVMCBFUTJK56TCFB4ABCXQQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.112648534 (445h13m55.983731579s) [noise=true , address=145.79.227.13:4804 , piece_id=L4LQOMARXANONKYYKIUKFH3H25CJ6XFAQ2D5YVILKHXHHFKGHP7A, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.129221522 (445h13m56.000304567s) [noise=true , address=145.79.227.50:4373 , piece_id=EDQNNC2DIWQTGGEA7KT6BPDJGIAHT42IQZPZ7YVC3FDHTASZ3WQQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.135066673 (445h13m56.006149708s) [noise=true , address=145.79.227.39:4013 , piece_id=7KCQCZV7IS532ADB4DLESWHRUSB5ZWIZ25HW5YNWSFXWQB4TROSQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.136122593 (445h13m56.007205628s) [noise=true , address=145.79.227.32:4834 , piece_id=CIY75P6ZSW5VKRMENI7AQA3PSYBACHVKGB4XAFQYKTDBZHT7AXRQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.297639110 (445h13m56.168722145s) [noise=true , address=77.90.31.16:4736 , piece_id=E4BFE7Q4WTUPROYTG7IR4QLM4LL5XKCDS442KO654PB6BLEACSFQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.327366527 (445h13m56.198449562s) [noise=true , address=45.91.116.4:4939 , piece_id=DRA22PE2YDC3RYWT65F2POHKV47HEV22IGSYC5IPDVX2PQ7WDP2A, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.353346166 (445h13m56.224429211s) [noise=true , address=77.90.31.14:4661 , piece_id=TODCW4FPSPAGVDWRIASF3PKOLXUNHPWSC62QPAWKSGYFNH4UKDEQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.362861025 (445h13m56.23394406s) [noise=true , address=45.91.116.7:4010 , piece_id=KNG2IB4WHOMFXZM42EI25ML3EX6IPNWSXMDYTJUZMQYHCAW7YYWA, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.373991459 (445h13m56.245074504s) [noise=true , address=77.90.31.12:4590 , piece_id=J74VB7H77KUHJQGQLD2SDLRPYVPPKIFFQL6ZZTNYZCBTP7ZNGXAQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.463096260 (445h13m56.334179305s) [noise=true , address=45.91.116.6:4974 , piece_id=DDIMQY46RCO5U3G3ZKWIIXAV44TUGRK5YTNVAF5OQPAKM5TJ637Q, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:43:23.169031756 (445h14m23.040114801s) [noise=true , address=145.79.227.40:4049 , piece_id=FODTLUWOIDSDZA5QFJ5RZPSYHRMPFOLK5NJCRUZ7GIXC236IRDQQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:39362->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:39362->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39362->145.79.227.40:4049: use of closed network connection Dec 24 04:43:23.169171592 (445h14m23.040254637s) [noise=true , address=145.79.227.51:4410 , piece_id=2CEZTRDLXVAXFFEAEOF5SVHSOXRVB5ZTSQXYAN5GD4Y6ZU3DGV4Q, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX, address:145.79.227.51:4410): protocol: write tcp 188.166.225.170:40626->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:40626->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40626->145.79.227.51:4410: use of closed network connection Dec 24 04:43:23.169262826 (445h14m23.040345872s) [noise=true , address=145.79.227.24:4581 , piece_id=RKCKZQ7IXVKJ6WMQ7P2AQWTW5GYORBLFXAMTS7YBVY6YFK7ERCRA, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:60164->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:60164->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60164->145.79.227.24:4581: use of closed network connection Dec 24 04:43:23.169311709 (445h14m23.040394754s) [seg_pos={0 0}, upload=390839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:43:23.405890860 (445h14m23.276973895s) [upload=390839]: Done uploading segments... Dec 24 04:43:23.405935244 (445h14m23.277018279s) [upload=390839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.896498787s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.813773293s trace: https://jaeger.dev.storj.io/trace/64b51f8bd213f167 FAIL (2025-12-24 04:48:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:42:53.854141273 (445h13m53.725224318s) [upload=390839]: Starting upload Dec 24 04:42:53.854215494 (445h13m53.725298539s) [upload=390839]: Done starting upload Dec 24 04:42:53.854479075 (445h13m53.72556212s) [upload=390839]: Uploading segments... Dec 24 04:42:53.854577853 (445h13m53.725660908s) [upload=390839]: Got next segment. Inline: false Dec 24 04:42:53.854618741 (445h13m53.725701776s) [upload=390839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:42:53.854626226 (445h13m53.725709261s) [upload=390839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:42:54.098954572 (445h13m53.970037617s) [seg_pos={0 0}, upload=390839]: Begin upload segment... Dec 24 04:42:54.099000660 (445h13m53.970083695s) [seg_pos={0 0}, upload=390839]: Done begin upload segment. Dec 24 04:42:54.099029364 (445h13m53.970112409s) [noise=true , address=145.79.227.39:4013 , piece_id=7KCQCZV7IS532ADB4DLESWHRUSB5ZWIZ25HW5YNWSFXWQB4TROSQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.099140977 (445h13m53.970224022s) [noise=true , address=77.90.31.14:4661 , piece_id=TODCW4FPSPAGVDWRIASF3PKOLXUNHPWSC62QPAWKSGYFNH4UKDEQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.099586515 (445h13m53.97066955s) [noise=true , address=145.79.227.46:4602 , piece_id=XNUM44DBNEMIIEYQ66MMOZSHIWN2O5QYV4LHK2452BP5O6EOST4A, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.099863232 (445h13m53.970946267s) [noise=true , address=145.79.227.50:4373 , piece_id=EDQNNC2DIWQTGGEA7KT6BPDJGIAHT42IQZPZ7YVC3FDHTASZ3WQQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.100147754 (445h13m53.971230799s) [noise=true , address=45.91.116.4:4939 , piece_id=DRA22PE2YDC3RYWT65F2POHKV47HEV22IGSYC5IPDVX2PQ7WDP2A, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.100455209 (445h13m53.971538244s) [noise=true , address=45.91.116.7:4010 , piece_id=KNG2IB4WHOMFXZM42EI25ML3EX6IPNWSXMDYTJUZMQYHCAW7YYWA, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.100748137 (445h13m53.971831172s) [noise=true , address=145.79.227.51:4410 , piece_id=2CEZTRDLXVAXFFEAEOF5SVHSOXRVB5ZTSQXYAN5GD4Y6ZU3DGV4Q, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.101029513 (445h13m53.972112548s) [noise=true , address=45.91.116.6:4974 , piece_id=DDIMQY46RCO5U3G3ZKWIIXAV44TUGRK5YTNVAF5OQPAKM5TJ637Q, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.101365643 (445h13m53.972448678s) [noise=true , address=145.79.227.40:4049 , piece_id=FODTLUWOIDSDZA5QFJ5RZPSYHRMPFOLK5NJCRUZ7GIXC236IRDQQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.101958713 (445h13m53.973041748s) [noise=true , address=77.90.31.19:4841 , piece_id=EUONIBKTAQGXYM6CAQVZ3FJWHJQAOVFBOCCF4XYLTQI4IU566XTQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.102487159 (445h13m53.973570204s) [noise=true , address=145.79.227.32:4834 , piece_id=CIY75P6ZSW5VKRMENI7AQA3PSYBACHVKGB4XAFQYKTDBZHT7AXRQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.102824290 (445h13m53.973907325s) [noise=true , address=77.90.31.12:4590 , piece_id=J74VB7H77KUHJQGQLD2SDLRPYVPPKIFFQL6ZZTNYZCBTP7ZNGXAQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.103165290 (445h13m53.974248325s) [noise=true , address=77.90.31.16:4736 , piece_id=E4BFE7Q4WTUPROYTG7IR4QLM4LL5XKCDS442KO654PB6BLEACSFQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.104097395 (445h13m53.97518043s) [noise=true , address=145.79.227.13:4804 , piece_id=L4LQOMARXANONKYYKIUKFH3H25CJ6XFAQ2D5YVILKHXHHFKGHP7A, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.106264812 (445h13m53.977347847s) [noise=true , address=145.79.227.24:4581 , piece_id=RKCKZQ7IXVKJ6WMQ7P2AQWTW5GYORBLFXAMTS7YBVY6YFK7ERCRA, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.107835482 (445h13m53.978918517s) [noise=true , address=145.79.227.49:4339 , piece_id=OE3ELR2OHWI27NPIY6C6HI2C6DK6LIVMCBFUTJK56TCFB4ABCXQQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:55.256583228 (445h13m55.127666264s) [noise=true , address=77.90.31.19:4841 , piece_id=EUONIBKTAQGXYM6CAQVZ3FJWHJQAOVFBOCCF4XYLTQI4IU566XTQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 24 04:42:55.825208676 (445h13m55.696291711s) [upload=390839]: Next returned nil segment Dec 24 04:42:55.825231309 (445h13m55.696314354s) [upload=390839]: Waiting for error group managing segments... Dec 24 04:42:56.111380959 (445h13m55.982463994s) [noise=true , address=145.79.227.46:4602 , piece_id=XNUM44DBNEMIIEYQ66MMOZSHIWN2O5QYV4LHK2452BP5O6EOST4A, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.112285412 (445h13m55.983368457s) [noise=true , address=145.79.227.49:4339 , piece_id=OE3ELR2OHWI27NPIY6C6HI2C6DK6LIVMCBFUTJK56TCFB4ABCXQQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.112648534 (445h13m55.983731579s) [noise=true , address=145.79.227.13:4804 , piece_id=L4LQOMARXANONKYYKIUKFH3H25CJ6XFAQ2D5YVILKHXHHFKGHP7A, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.129221522 (445h13m56.000304567s) [noise=true , address=145.79.227.50:4373 , piece_id=EDQNNC2DIWQTGGEA7KT6BPDJGIAHT42IQZPZ7YVC3FDHTASZ3WQQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.135066673 (445h13m56.006149708s) [noise=true , address=145.79.227.39:4013 , piece_id=7KCQCZV7IS532ADB4DLESWHRUSB5ZWIZ25HW5YNWSFXWQB4TROSQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.136122593 (445h13m56.007205628s) [noise=true , address=145.79.227.32:4834 , piece_id=CIY75P6ZSW5VKRMENI7AQA3PSYBACHVKGB4XAFQYKTDBZHT7AXRQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.297639110 (445h13m56.168722145s) [noise=true , address=77.90.31.16:4736 , piece_id=E4BFE7Q4WTUPROYTG7IR4QLM4LL5XKCDS442KO654PB6BLEACSFQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.327366527 (445h13m56.198449562s) [noise=true , address=45.91.116.4:4939 , piece_id=DRA22PE2YDC3RYWT65F2POHKV47HEV22IGSYC5IPDVX2PQ7WDP2A, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.353346166 (445h13m56.224429211s) [noise=true , address=77.90.31.14:4661 , piece_id=TODCW4FPSPAGVDWRIASF3PKOLXUNHPWSC62QPAWKSGYFNH4UKDEQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.362861025 (445h13m56.23394406s) [noise=true , address=45.91.116.7:4010 , piece_id=KNG2IB4WHOMFXZM42EI25ML3EX6IPNWSXMDYTJUZMQYHCAW7YYWA, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.373991459 (445h13m56.245074504s) [noise=true , address=77.90.31.12:4590 , piece_id=J74VB7H77KUHJQGQLD2SDLRPYVPPKIFFQL6ZZTNYZCBTP7ZNGXAQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.463096260 (445h13m56.334179305s) [noise=true , address=45.91.116.6:4974 , piece_id=DDIMQY46RCO5U3G3ZKWIIXAV44TUGRK5YTNVAF5OQPAKM5TJ637Q, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:43:23.169031756 (445h14m23.040114801s) [noise=true , address=145.79.227.40:4049 , piece_id=FODTLUWOIDSDZA5QFJ5RZPSYHRMPFOLK5NJCRUZ7GIXC236IRDQQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:39362->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:39362->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39362->145.79.227.40:4049: use of closed network connection Dec 24 04:43:23.169171592 (445h14m23.040254637s) [noise=true , address=145.79.227.51:4410 , piece_id=2CEZTRDLXVAXFFEAEOF5SVHSOXRVB5ZTSQXYAN5GD4Y6ZU3DGV4Q, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX, address:145.79.227.51:4410): protocol: write tcp 188.166.225.170:40626->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:40626->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40626->145.79.227.51:4410: use of closed network connection Dec 24 04:43:23.169262826 (445h14m23.040345872s) [noise=true , address=145.79.227.24:4581 , piece_id=RKCKZQ7IXVKJ6WMQ7P2AQWTW5GYORBLFXAMTS7YBVY6YFK7ERCRA, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:60164->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:60164->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60164->145.79.227.24:4581: use of closed network connection Dec 24 04:43:23.169311709 (445h14m23.040394754s) [seg_pos={0 0}, upload=390839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:43:23.405890860 (445h14m23.276973895s) [upload=390839]: Done uploading segments... Dec 24 04:43:23.405935244 (445h14m23.277018279s) [upload=390839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.813801005s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.965951099s trace: https://jaeger.dev.storj.io/trace/64b51f8bd213f167 FAIL (2025-12-24 04:47:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:42:53.854141273 (445h13m53.725224318s) [upload=390839]: Starting upload Dec 24 04:42:53.854215494 (445h13m53.725298539s) [upload=390839]: Done starting upload Dec 24 04:42:53.854479075 (445h13m53.72556212s) [upload=390839]: Uploading segments... Dec 24 04:42:53.854577853 (445h13m53.725660908s) [upload=390839]: Got next segment. Inline: false Dec 24 04:42:53.854618741 (445h13m53.725701776s) [upload=390839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:42:53.854626226 (445h13m53.725709261s) [upload=390839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:42:54.098954572 (445h13m53.970037617s) [seg_pos={0 0}, upload=390839]: Begin upload segment... Dec 24 04:42:54.099000660 (445h13m53.970083695s) [seg_pos={0 0}, upload=390839]: Done begin upload segment. Dec 24 04:42:54.099029364 (445h13m53.970112409s) [noise=true , address=145.79.227.39:4013 , piece_id=7KCQCZV7IS532ADB4DLESWHRUSB5ZWIZ25HW5YNWSFXWQB4TROSQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.099140977 (445h13m53.970224022s) [noise=true , address=77.90.31.14:4661 , piece_id=TODCW4FPSPAGVDWRIASF3PKOLXUNHPWSC62QPAWKSGYFNH4UKDEQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.099586515 (445h13m53.97066955s) [noise=true , address=145.79.227.46:4602 , piece_id=XNUM44DBNEMIIEYQ66MMOZSHIWN2O5QYV4LHK2452BP5O6EOST4A, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.099863232 (445h13m53.970946267s) [noise=true , address=145.79.227.50:4373 , piece_id=EDQNNC2DIWQTGGEA7KT6BPDJGIAHT42IQZPZ7YVC3FDHTASZ3WQQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.100147754 (445h13m53.971230799s) [noise=true , address=45.91.116.4:4939 , piece_id=DRA22PE2YDC3RYWT65F2POHKV47HEV22IGSYC5IPDVX2PQ7WDP2A, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.100455209 (445h13m53.971538244s) [noise=true , address=45.91.116.7:4010 , piece_id=KNG2IB4WHOMFXZM42EI25ML3EX6IPNWSXMDYTJUZMQYHCAW7YYWA, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.100748137 (445h13m53.971831172s) [noise=true , address=145.79.227.51:4410 , piece_id=2CEZTRDLXVAXFFEAEOF5SVHSOXRVB5ZTSQXYAN5GD4Y6ZU3DGV4Q, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.101029513 (445h13m53.972112548s) [noise=true , address=45.91.116.6:4974 , piece_id=DDIMQY46RCO5U3G3ZKWIIXAV44TUGRK5YTNVAF5OQPAKM5TJ637Q, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.101365643 (445h13m53.972448678s) [noise=true , address=145.79.227.40:4049 , piece_id=FODTLUWOIDSDZA5QFJ5RZPSYHRMPFOLK5NJCRUZ7GIXC236IRDQQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.101958713 (445h13m53.973041748s) [noise=true , address=77.90.31.19:4841 , piece_id=EUONIBKTAQGXYM6CAQVZ3FJWHJQAOVFBOCCF4XYLTQI4IU566XTQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.102487159 (445h13m53.973570204s) [noise=true , address=145.79.227.32:4834 , piece_id=CIY75P6ZSW5VKRMENI7AQA3PSYBACHVKGB4XAFQYKTDBZHT7AXRQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.102824290 (445h13m53.973907325s) [noise=true , address=77.90.31.12:4590 , piece_id=J74VB7H77KUHJQGQLD2SDLRPYVPPKIFFQL6ZZTNYZCBTP7ZNGXAQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.103165290 (445h13m53.974248325s) [noise=true , address=77.90.31.16:4736 , piece_id=E4BFE7Q4WTUPROYTG7IR4QLM4LL5XKCDS442KO654PB6BLEACSFQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.104097395 (445h13m53.97518043s) [noise=true , address=145.79.227.13:4804 , piece_id=L4LQOMARXANONKYYKIUKFH3H25CJ6XFAQ2D5YVILKHXHHFKGHP7A, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.106264812 (445h13m53.977347847s) [noise=true , address=145.79.227.24:4581 , piece_id=RKCKZQ7IXVKJ6WMQ7P2AQWTW5GYORBLFXAMTS7YBVY6YFK7ERCRA, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.107835482 (445h13m53.978918517s) [noise=true , address=145.79.227.49:4339 , piece_id=OE3ELR2OHWI27NPIY6C6HI2C6DK6LIVMCBFUTJK56TCFB4ABCXQQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:55.256583228 (445h13m55.127666264s) [noise=true , address=77.90.31.19:4841 , piece_id=EUONIBKTAQGXYM6CAQVZ3FJWHJQAOVFBOCCF4XYLTQI4IU566XTQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 24 04:42:55.825208676 (445h13m55.696291711s) [upload=390839]: Next returned nil segment Dec 24 04:42:55.825231309 (445h13m55.696314354s) [upload=390839]: Waiting for error group managing segments... Dec 24 04:42:56.111380959 (445h13m55.982463994s) [noise=true , address=145.79.227.46:4602 , piece_id=XNUM44DBNEMIIEYQ66MMOZSHIWN2O5QYV4LHK2452BP5O6EOST4A, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.112285412 (445h13m55.983368457s) [noise=true , address=145.79.227.49:4339 , piece_id=OE3ELR2OHWI27NPIY6C6HI2C6DK6LIVMCBFUTJK56TCFB4ABCXQQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.112648534 (445h13m55.983731579s) [noise=true , address=145.79.227.13:4804 , piece_id=L4LQOMARXANONKYYKIUKFH3H25CJ6XFAQ2D5YVILKHXHHFKGHP7A, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.129221522 (445h13m56.000304567s) [noise=true , address=145.79.227.50:4373 , piece_id=EDQNNC2DIWQTGGEA7KT6BPDJGIAHT42IQZPZ7YVC3FDHTASZ3WQQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.135066673 (445h13m56.006149708s) [noise=true , address=145.79.227.39:4013 , piece_id=7KCQCZV7IS532ADB4DLESWHRUSB5ZWIZ25HW5YNWSFXWQB4TROSQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.136122593 (445h13m56.007205628s) [noise=true , address=145.79.227.32:4834 , piece_id=CIY75P6ZSW5VKRMENI7AQA3PSYBACHVKGB4XAFQYKTDBZHT7AXRQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.297639110 (445h13m56.168722145s) [noise=true , address=77.90.31.16:4736 , piece_id=E4BFE7Q4WTUPROYTG7IR4QLM4LL5XKCDS442KO654PB6BLEACSFQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.327366527 (445h13m56.198449562s) [noise=true , address=45.91.116.4:4939 , piece_id=DRA22PE2YDC3RYWT65F2POHKV47HEV22IGSYC5IPDVX2PQ7WDP2A, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.353346166 (445h13m56.224429211s) [noise=true , address=77.90.31.14:4661 , piece_id=TODCW4FPSPAGVDWRIASF3PKOLXUNHPWSC62QPAWKSGYFNH4UKDEQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.362861025 (445h13m56.23394406s) [noise=true , address=45.91.116.7:4010 , piece_id=KNG2IB4WHOMFXZM42EI25ML3EX6IPNWSXMDYTJUZMQYHCAW7YYWA, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.373991459 (445h13m56.245074504s) [noise=true , address=77.90.31.12:4590 , piece_id=J74VB7H77KUHJQGQLD2SDLRPYVPPKIFFQL6ZZTNYZCBTP7ZNGXAQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.463096260 (445h13m56.334179305s) [noise=true , address=45.91.116.6:4974 , piece_id=DDIMQY46RCO5U3G3ZKWIIXAV44TUGRK5YTNVAF5OQPAKM5TJ637Q, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:43:23.169031756 (445h14m23.040114801s) [noise=true , address=145.79.227.40:4049 , piece_id=FODTLUWOIDSDZA5QFJ5RZPSYHRMPFOLK5NJCRUZ7GIXC236IRDQQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:39362->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:39362->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39362->145.79.227.40:4049: use of closed network connection Dec 24 04:43:23.169171592 (445h14m23.040254637s) [noise=true , address=145.79.227.51:4410 , piece_id=2CEZTRDLXVAXFFEAEOF5SVHSOXRVB5ZTSQXYAN5GD4Y6ZU3DGV4Q, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX, address:145.79.227.51:4410): protocol: write tcp 188.166.225.170:40626->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:40626->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40626->145.79.227.51:4410: use of closed network connection Dec 24 04:43:23.169262826 (445h14m23.040345872s) [noise=true , address=145.79.227.24:4581 , piece_id=RKCKZQ7IXVKJ6WMQ7P2AQWTW5GYORBLFXAMTS7YBVY6YFK7ERCRA, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:60164->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:60164->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60164->145.79.227.24:4581: use of closed network connection Dec 24 04:43:23.169311709 (445h14m23.040394754s) [seg_pos={0 0}, upload=390839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:43:23.405890860 (445h14m23.276973895s) [upload=390839]: Done uploading segments... Dec 24 04:43:23.405935244 (445h14m23.277018279s) [upload=390839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.965970695s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 8.336659359s trace: https://jaeger.dev.storj.io/trace/64b51f8bd213f167 FAIL (2025-12-24 04:47:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:42:53.854141273 (445h13m53.725224318s) [upload=390839]: Starting upload Dec 24 04:42:53.854215494 (445h13m53.725298539s) [upload=390839]: Done starting upload Dec 24 04:42:53.854479075 (445h13m53.72556212s) [upload=390839]: Uploading segments... Dec 24 04:42:53.854577853 (445h13m53.725660908s) [upload=390839]: Got next segment. Inline: false Dec 24 04:42:53.854618741 (445h13m53.725701776s) [upload=390839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:42:53.854626226 (445h13m53.725709261s) [upload=390839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:42:54.098954572 (445h13m53.970037617s) [seg_pos={0 0}, upload=390839]: Begin upload segment... Dec 24 04:42:54.099000660 (445h13m53.970083695s) [seg_pos={0 0}, upload=390839]: Done begin upload segment. Dec 24 04:42:54.099029364 (445h13m53.970112409s) [noise=true , address=145.79.227.39:4013 , piece_id=7KCQCZV7IS532ADB4DLESWHRUSB5ZWIZ25HW5YNWSFXWQB4TROSQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.099140977 (445h13m53.970224022s) [noise=true , address=77.90.31.14:4661 , piece_id=TODCW4FPSPAGVDWRIASF3PKOLXUNHPWSC62QPAWKSGYFNH4UKDEQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.099586515 (445h13m53.97066955s) [noise=true , address=145.79.227.46:4602 , piece_id=XNUM44DBNEMIIEYQ66MMOZSHIWN2O5QYV4LHK2452BP5O6EOST4A, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.099863232 (445h13m53.970946267s) [noise=true , address=145.79.227.50:4373 , piece_id=EDQNNC2DIWQTGGEA7KT6BPDJGIAHT42IQZPZ7YVC3FDHTASZ3WQQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.100147754 (445h13m53.971230799s) [noise=true , address=45.91.116.4:4939 , piece_id=DRA22PE2YDC3RYWT65F2POHKV47HEV22IGSYC5IPDVX2PQ7WDP2A, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.100455209 (445h13m53.971538244s) [noise=true , address=45.91.116.7:4010 , piece_id=KNG2IB4WHOMFXZM42EI25ML3EX6IPNWSXMDYTJUZMQYHCAW7YYWA, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.100748137 (445h13m53.971831172s) [noise=true , address=145.79.227.51:4410 , piece_id=2CEZTRDLXVAXFFEAEOF5SVHSOXRVB5ZTSQXYAN5GD4Y6ZU3DGV4Q, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.101029513 (445h13m53.972112548s) [noise=true , address=45.91.116.6:4974 , piece_id=DDIMQY46RCO5U3G3ZKWIIXAV44TUGRK5YTNVAF5OQPAKM5TJ637Q, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.101365643 (445h13m53.972448678s) [noise=true , address=145.79.227.40:4049 , piece_id=FODTLUWOIDSDZA5QFJ5RZPSYHRMPFOLK5NJCRUZ7GIXC236IRDQQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.101958713 (445h13m53.973041748s) [noise=true , address=77.90.31.19:4841 , piece_id=EUONIBKTAQGXYM6CAQVZ3FJWHJQAOVFBOCCF4XYLTQI4IU566XTQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.102487159 (445h13m53.973570204s) [noise=true , address=145.79.227.32:4834 , piece_id=CIY75P6ZSW5VKRMENI7AQA3PSYBACHVKGB4XAFQYKTDBZHT7AXRQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.102824290 (445h13m53.973907325s) [noise=true , address=77.90.31.12:4590 , piece_id=J74VB7H77KUHJQGQLD2SDLRPYVPPKIFFQL6ZZTNYZCBTP7ZNGXAQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.103165290 (445h13m53.974248325s) [noise=true , address=77.90.31.16:4736 , piece_id=E4BFE7Q4WTUPROYTG7IR4QLM4LL5XKCDS442KO654PB6BLEACSFQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.104097395 (445h13m53.97518043s) [noise=true , address=145.79.227.13:4804 , piece_id=L4LQOMARXANONKYYKIUKFH3H25CJ6XFAQ2D5YVILKHXHHFKGHP7A, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.106264812 (445h13m53.977347847s) [noise=true , address=145.79.227.24:4581 , piece_id=RKCKZQ7IXVKJ6WMQ7P2AQWTW5GYORBLFXAMTS7YBVY6YFK7ERCRA, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.107835482 (445h13m53.978918517s) [noise=true , address=145.79.227.49:4339 , piece_id=OE3ELR2OHWI27NPIY6C6HI2C6DK6LIVMCBFUTJK56TCFB4ABCXQQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:55.256583228 (445h13m55.127666264s) [noise=true , address=77.90.31.19:4841 , piece_id=EUONIBKTAQGXYM6CAQVZ3FJWHJQAOVFBOCCF4XYLTQI4IU566XTQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 24 04:42:55.825208676 (445h13m55.696291711s) [upload=390839]: Next returned nil segment Dec 24 04:42:55.825231309 (445h13m55.696314354s) [upload=390839]: Waiting for error group managing segments... Dec 24 04:42:56.111380959 (445h13m55.982463994s) [noise=true , address=145.79.227.46:4602 , piece_id=XNUM44DBNEMIIEYQ66MMOZSHIWN2O5QYV4LHK2452BP5O6EOST4A, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.112285412 (445h13m55.983368457s) [noise=true , address=145.79.227.49:4339 , piece_id=OE3ELR2OHWI27NPIY6C6HI2C6DK6LIVMCBFUTJK56TCFB4ABCXQQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.112648534 (445h13m55.983731579s) [noise=true , address=145.79.227.13:4804 , piece_id=L4LQOMARXANONKYYKIUKFH3H25CJ6XFAQ2D5YVILKHXHHFKGHP7A, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.129221522 (445h13m56.000304567s) [noise=true , address=145.79.227.50:4373 , piece_id=EDQNNC2DIWQTGGEA7KT6BPDJGIAHT42IQZPZ7YVC3FDHTASZ3WQQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.135066673 (445h13m56.006149708s) [noise=true , address=145.79.227.39:4013 , piece_id=7KCQCZV7IS532ADB4DLESWHRUSB5ZWIZ25HW5YNWSFXWQB4TROSQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.136122593 (445h13m56.007205628s) [noise=true , address=145.79.227.32:4834 , piece_id=CIY75P6ZSW5VKRMENI7AQA3PSYBACHVKGB4XAFQYKTDBZHT7AXRQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.297639110 (445h13m56.168722145s) [noise=true , address=77.90.31.16:4736 , piece_id=E4BFE7Q4WTUPROYTG7IR4QLM4LL5XKCDS442KO654PB6BLEACSFQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.327366527 (445h13m56.198449562s) [noise=true , address=45.91.116.4:4939 , piece_id=DRA22PE2YDC3RYWT65F2POHKV47HEV22IGSYC5IPDVX2PQ7WDP2A, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.353346166 (445h13m56.224429211s) [noise=true , address=77.90.31.14:4661 , piece_id=TODCW4FPSPAGVDWRIASF3PKOLXUNHPWSC62QPAWKSGYFNH4UKDEQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.362861025 (445h13m56.23394406s) [noise=true , address=45.91.116.7:4010 , piece_id=KNG2IB4WHOMFXZM42EI25ML3EX6IPNWSXMDYTJUZMQYHCAW7YYWA, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.373991459 (445h13m56.245074504s) [noise=true , address=77.90.31.12:4590 , piece_id=J74VB7H77KUHJQGQLD2SDLRPYVPPKIFFQL6ZZTNYZCBTP7ZNGXAQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.463096260 (445h13m56.334179305s) [noise=true , address=45.91.116.6:4974 , piece_id=DDIMQY46RCO5U3G3ZKWIIXAV44TUGRK5YTNVAF5OQPAKM5TJ637Q, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:43:23.169031756 (445h14m23.040114801s) [noise=true , address=145.79.227.40:4049 , piece_id=FODTLUWOIDSDZA5QFJ5RZPSYHRMPFOLK5NJCRUZ7GIXC236IRDQQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:39362->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:39362->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39362->145.79.227.40:4049: use of closed network connection Dec 24 04:43:23.169171592 (445h14m23.040254637s) [noise=true , address=145.79.227.51:4410 , piece_id=2CEZTRDLXVAXFFEAEOF5SVHSOXRVB5ZTSQXYAN5GD4Y6ZU3DGV4Q, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX, address:145.79.227.51:4410): protocol: write tcp 188.166.225.170:40626->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:40626->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40626->145.79.227.51:4410: use of closed network connection Dec 24 04:43:23.169262826 (445h14m23.040345872s) [noise=true , address=145.79.227.24:4581 , piece_id=RKCKZQ7IXVKJ6WMQ7P2AQWTW5GYORBLFXAMTS7YBVY6YFK7ERCRA, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:60164->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:60164->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60164->145.79.227.24:4581: use of closed network connection Dec 24 04:43:23.169311709 (445h14m23.040394754s) [seg_pos={0 0}, upload=390839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:43:23.405890860 (445h14m23.276973895s) [upload=390839]: Done uploading segments... Dec 24 04:43:23.405935244 (445h14m23.277018279s) [upload=390839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 8.336688504s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.785357865s trace: https://jaeger.dev.storj.io/trace/64b51f8bd213f167 FAIL (2025-12-24 04:46:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:42:53.854141273 (445h13m53.725224318s) [upload=390839]: Starting upload Dec 24 04:42:53.854215494 (445h13m53.725298539s) [upload=390839]: Done starting upload Dec 24 04:42:53.854479075 (445h13m53.72556212s) [upload=390839]: Uploading segments... Dec 24 04:42:53.854577853 (445h13m53.725660908s) [upload=390839]: Got next segment. Inline: false Dec 24 04:42:53.854618741 (445h13m53.725701776s) [upload=390839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:42:53.854626226 (445h13m53.725709261s) [upload=390839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:42:54.098954572 (445h13m53.970037617s) [seg_pos={0 0}, upload=390839]: Begin upload segment... Dec 24 04:42:54.099000660 (445h13m53.970083695s) [seg_pos={0 0}, upload=390839]: Done begin upload segment. Dec 24 04:42:54.099029364 (445h13m53.970112409s) [noise=true , address=145.79.227.39:4013 , piece_id=7KCQCZV7IS532ADB4DLESWHRUSB5ZWIZ25HW5YNWSFXWQB4TROSQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.099140977 (445h13m53.970224022s) [noise=true , address=77.90.31.14:4661 , piece_id=TODCW4FPSPAGVDWRIASF3PKOLXUNHPWSC62QPAWKSGYFNH4UKDEQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.099586515 (445h13m53.97066955s) [noise=true , address=145.79.227.46:4602 , piece_id=XNUM44DBNEMIIEYQ66MMOZSHIWN2O5QYV4LHK2452BP5O6EOST4A, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.099863232 (445h13m53.970946267s) [noise=true , address=145.79.227.50:4373 , piece_id=EDQNNC2DIWQTGGEA7KT6BPDJGIAHT42IQZPZ7YVC3FDHTASZ3WQQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.100147754 (445h13m53.971230799s) [noise=true , address=45.91.116.4:4939 , piece_id=DRA22PE2YDC3RYWT65F2POHKV47HEV22IGSYC5IPDVX2PQ7WDP2A, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.100455209 (445h13m53.971538244s) [noise=true , address=45.91.116.7:4010 , piece_id=KNG2IB4WHOMFXZM42EI25ML3EX6IPNWSXMDYTJUZMQYHCAW7YYWA, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.100748137 (445h13m53.971831172s) [noise=true , address=145.79.227.51:4410 , piece_id=2CEZTRDLXVAXFFEAEOF5SVHSOXRVB5ZTSQXYAN5GD4Y6ZU3DGV4Q, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.101029513 (445h13m53.972112548s) [noise=true , address=45.91.116.6:4974 , piece_id=DDIMQY46RCO5U3G3ZKWIIXAV44TUGRK5YTNVAF5OQPAKM5TJ637Q, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.101365643 (445h13m53.972448678s) [noise=true , address=145.79.227.40:4049 , piece_id=FODTLUWOIDSDZA5QFJ5RZPSYHRMPFOLK5NJCRUZ7GIXC236IRDQQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.101958713 (445h13m53.973041748s) [noise=true , address=77.90.31.19:4841 , piece_id=EUONIBKTAQGXYM6CAQVZ3FJWHJQAOVFBOCCF4XYLTQI4IU566XTQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.102487159 (445h13m53.973570204s) [noise=true , address=145.79.227.32:4834 , piece_id=CIY75P6ZSW5VKRMENI7AQA3PSYBACHVKGB4XAFQYKTDBZHT7AXRQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.102824290 (445h13m53.973907325s) [noise=true , address=77.90.31.12:4590 , piece_id=J74VB7H77KUHJQGQLD2SDLRPYVPPKIFFQL6ZZTNYZCBTP7ZNGXAQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.103165290 (445h13m53.974248325s) [noise=true , address=77.90.31.16:4736 , piece_id=E4BFE7Q4WTUPROYTG7IR4QLM4LL5XKCDS442KO654PB6BLEACSFQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.104097395 (445h13m53.97518043s) [noise=true , address=145.79.227.13:4804 , piece_id=L4LQOMARXANONKYYKIUKFH3H25CJ6XFAQ2D5YVILKHXHHFKGHP7A, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.106264812 (445h13m53.977347847s) [noise=true , address=145.79.227.24:4581 , piece_id=RKCKZQ7IXVKJ6WMQ7P2AQWTW5GYORBLFXAMTS7YBVY6YFK7ERCRA, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.107835482 (445h13m53.978918517s) [noise=true , address=145.79.227.49:4339 , piece_id=OE3ELR2OHWI27NPIY6C6HI2C6DK6LIVMCBFUTJK56TCFB4ABCXQQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:55.256583228 (445h13m55.127666264s) [noise=true , address=77.90.31.19:4841 , piece_id=EUONIBKTAQGXYM6CAQVZ3FJWHJQAOVFBOCCF4XYLTQI4IU566XTQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 24 04:42:55.825208676 (445h13m55.696291711s) [upload=390839]: Next returned nil segment Dec 24 04:42:55.825231309 (445h13m55.696314354s) [upload=390839]: Waiting for error group managing segments... Dec 24 04:42:56.111380959 (445h13m55.982463994s) [noise=true , address=145.79.227.46:4602 , piece_id=XNUM44DBNEMIIEYQ66MMOZSHIWN2O5QYV4LHK2452BP5O6EOST4A, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.112285412 (445h13m55.983368457s) [noise=true , address=145.79.227.49:4339 , piece_id=OE3ELR2OHWI27NPIY6C6HI2C6DK6LIVMCBFUTJK56TCFB4ABCXQQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.112648534 (445h13m55.983731579s) [noise=true , address=145.79.227.13:4804 , piece_id=L4LQOMARXANONKYYKIUKFH3H25CJ6XFAQ2D5YVILKHXHHFKGHP7A, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.129221522 (445h13m56.000304567s) [noise=true , address=145.79.227.50:4373 , piece_id=EDQNNC2DIWQTGGEA7KT6BPDJGIAHT42IQZPZ7YVC3FDHTASZ3WQQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.135066673 (445h13m56.006149708s) [noise=true , address=145.79.227.39:4013 , piece_id=7KCQCZV7IS532ADB4DLESWHRUSB5ZWIZ25HW5YNWSFXWQB4TROSQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.136122593 (445h13m56.007205628s) [noise=true , address=145.79.227.32:4834 , piece_id=CIY75P6ZSW5VKRMENI7AQA3PSYBACHVKGB4XAFQYKTDBZHT7AXRQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.297639110 (445h13m56.168722145s) [noise=true , address=77.90.31.16:4736 , piece_id=E4BFE7Q4WTUPROYTG7IR4QLM4LL5XKCDS442KO654PB6BLEACSFQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.327366527 (445h13m56.198449562s) [noise=true , address=45.91.116.4:4939 , piece_id=DRA22PE2YDC3RYWT65F2POHKV47HEV22IGSYC5IPDVX2PQ7WDP2A, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.353346166 (445h13m56.224429211s) [noise=true , address=77.90.31.14:4661 , piece_id=TODCW4FPSPAGVDWRIASF3PKOLXUNHPWSC62QPAWKSGYFNH4UKDEQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.362861025 (445h13m56.23394406s) [noise=true , address=45.91.116.7:4010 , piece_id=KNG2IB4WHOMFXZM42EI25ML3EX6IPNWSXMDYTJUZMQYHCAW7YYWA, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.373991459 (445h13m56.245074504s) [noise=true , address=77.90.31.12:4590 , piece_id=J74VB7H77KUHJQGQLD2SDLRPYVPPKIFFQL6ZZTNYZCBTP7ZNGXAQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.463096260 (445h13m56.334179305s) [noise=true , address=45.91.116.6:4974 , piece_id=DDIMQY46RCO5U3G3ZKWIIXAV44TUGRK5YTNVAF5OQPAKM5TJ637Q, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:43:23.169031756 (445h14m23.040114801s) [noise=true , address=145.79.227.40:4049 , piece_id=FODTLUWOIDSDZA5QFJ5RZPSYHRMPFOLK5NJCRUZ7GIXC236IRDQQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:39362->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:39362->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39362->145.79.227.40:4049: use of closed network connection Dec 24 04:43:23.169171592 (445h14m23.040254637s) [noise=true , address=145.79.227.51:4410 , piece_id=2CEZTRDLXVAXFFEAEOF5SVHSOXRVB5ZTSQXYAN5GD4Y6ZU3DGV4Q, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX, address:145.79.227.51:4410): protocol: write tcp 188.166.225.170:40626->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:40626->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40626->145.79.227.51:4410: use of closed network connection Dec 24 04:43:23.169262826 (445h14m23.040345872s) [noise=true , address=145.79.227.24:4581 , piece_id=RKCKZQ7IXVKJ6WMQ7P2AQWTW5GYORBLFXAMTS7YBVY6YFK7ERCRA, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:60164->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:60164->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60164->145.79.227.24:4581: use of closed network connection Dec 24 04:43:23.169311709 (445h14m23.040394754s) [seg_pos={0 0}, upload=390839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:43:23.405890860 (445h14m23.276973895s) [upload=390839]: Done uploading segments... Dec 24 04:43:23.405935244 (445h14m23.277018279s) [upload=390839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.785413811s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.979710939s trace: https://jaeger.dev.storj.io/trace/64b51f8bd213f167 FAIL (2025-12-24 04:45:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:42:53.854141273 (445h13m53.725224318s) [upload=390839]: Starting upload Dec 24 04:42:53.854215494 (445h13m53.725298539s) [upload=390839]: Done starting upload Dec 24 04:42:53.854479075 (445h13m53.72556212s) [upload=390839]: Uploading segments... Dec 24 04:42:53.854577853 (445h13m53.725660908s) [upload=390839]: Got next segment. Inline: false Dec 24 04:42:53.854618741 (445h13m53.725701776s) [upload=390839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:42:53.854626226 (445h13m53.725709261s) [upload=390839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:42:54.098954572 (445h13m53.970037617s) [seg_pos={0 0}, upload=390839]: Begin upload segment... Dec 24 04:42:54.099000660 (445h13m53.970083695s) [seg_pos={0 0}, upload=390839]: Done begin upload segment. Dec 24 04:42:54.099029364 (445h13m53.970112409s) [noise=true , address=145.79.227.39:4013 , piece_id=7KCQCZV7IS532ADB4DLESWHRUSB5ZWIZ25HW5YNWSFXWQB4TROSQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.099140977 (445h13m53.970224022s) [noise=true , address=77.90.31.14:4661 , piece_id=TODCW4FPSPAGVDWRIASF3PKOLXUNHPWSC62QPAWKSGYFNH4UKDEQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.099586515 (445h13m53.97066955s) [noise=true , address=145.79.227.46:4602 , piece_id=XNUM44DBNEMIIEYQ66MMOZSHIWN2O5QYV4LHK2452BP5O6EOST4A, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.099863232 (445h13m53.970946267s) [noise=true , address=145.79.227.50:4373 , piece_id=EDQNNC2DIWQTGGEA7KT6BPDJGIAHT42IQZPZ7YVC3FDHTASZ3WQQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.100147754 (445h13m53.971230799s) [noise=true , address=45.91.116.4:4939 , piece_id=DRA22PE2YDC3RYWT65F2POHKV47HEV22IGSYC5IPDVX2PQ7WDP2A, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.100455209 (445h13m53.971538244s) [noise=true , address=45.91.116.7:4010 , piece_id=KNG2IB4WHOMFXZM42EI25ML3EX6IPNWSXMDYTJUZMQYHCAW7YYWA, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.100748137 (445h13m53.971831172s) [noise=true , address=145.79.227.51:4410 , piece_id=2CEZTRDLXVAXFFEAEOF5SVHSOXRVB5ZTSQXYAN5GD4Y6ZU3DGV4Q, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.101029513 (445h13m53.972112548s) [noise=true , address=45.91.116.6:4974 , piece_id=DDIMQY46RCO5U3G3ZKWIIXAV44TUGRK5YTNVAF5OQPAKM5TJ637Q, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.101365643 (445h13m53.972448678s) [noise=true , address=145.79.227.40:4049 , piece_id=FODTLUWOIDSDZA5QFJ5RZPSYHRMPFOLK5NJCRUZ7GIXC236IRDQQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.101958713 (445h13m53.973041748s) [noise=true , address=77.90.31.19:4841 , piece_id=EUONIBKTAQGXYM6CAQVZ3FJWHJQAOVFBOCCF4XYLTQI4IU566XTQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.102487159 (445h13m53.973570204s) [noise=true , address=145.79.227.32:4834 , piece_id=CIY75P6ZSW5VKRMENI7AQA3PSYBACHVKGB4XAFQYKTDBZHT7AXRQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.102824290 (445h13m53.973907325s) [noise=true , address=77.90.31.12:4590 , piece_id=J74VB7H77KUHJQGQLD2SDLRPYVPPKIFFQL6ZZTNYZCBTP7ZNGXAQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.103165290 (445h13m53.974248325s) [noise=true , address=77.90.31.16:4736 , piece_id=E4BFE7Q4WTUPROYTG7IR4QLM4LL5XKCDS442KO654PB6BLEACSFQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.104097395 (445h13m53.97518043s) [noise=true , address=145.79.227.13:4804 , piece_id=L4LQOMARXANONKYYKIUKFH3H25CJ6XFAQ2D5YVILKHXHHFKGHP7A, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.106264812 (445h13m53.977347847s) [noise=true , address=145.79.227.24:4581 , piece_id=RKCKZQ7IXVKJ6WMQ7P2AQWTW5GYORBLFXAMTS7YBVY6YFK7ERCRA, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.107835482 (445h13m53.978918517s) [noise=true , address=145.79.227.49:4339 , piece_id=OE3ELR2OHWI27NPIY6C6HI2C6DK6LIVMCBFUTJK56TCFB4ABCXQQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:55.256583228 (445h13m55.127666264s) [noise=true , address=77.90.31.19:4841 , piece_id=EUONIBKTAQGXYM6CAQVZ3FJWHJQAOVFBOCCF4XYLTQI4IU566XTQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 24 04:42:55.825208676 (445h13m55.696291711s) [upload=390839]: Next returned nil segment Dec 24 04:42:55.825231309 (445h13m55.696314354s) [upload=390839]: Waiting for error group managing segments... Dec 24 04:42:56.111380959 (445h13m55.982463994s) [noise=true , address=145.79.227.46:4602 , piece_id=XNUM44DBNEMIIEYQ66MMOZSHIWN2O5QYV4LHK2452BP5O6EOST4A, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.112285412 (445h13m55.983368457s) [noise=true , address=145.79.227.49:4339 , piece_id=OE3ELR2OHWI27NPIY6C6HI2C6DK6LIVMCBFUTJK56TCFB4ABCXQQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.112648534 (445h13m55.983731579s) [noise=true , address=145.79.227.13:4804 , piece_id=L4LQOMARXANONKYYKIUKFH3H25CJ6XFAQ2D5YVILKHXHHFKGHP7A, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.129221522 (445h13m56.000304567s) [noise=true , address=145.79.227.50:4373 , piece_id=EDQNNC2DIWQTGGEA7KT6BPDJGIAHT42IQZPZ7YVC3FDHTASZ3WQQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.135066673 (445h13m56.006149708s) [noise=true , address=145.79.227.39:4013 , piece_id=7KCQCZV7IS532ADB4DLESWHRUSB5ZWIZ25HW5YNWSFXWQB4TROSQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.136122593 (445h13m56.007205628s) [noise=true , address=145.79.227.32:4834 , piece_id=CIY75P6ZSW5VKRMENI7AQA3PSYBACHVKGB4XAFQYKTDBZHT7AXRQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.297639110 (445h13m56.168722145s) [noise=true , address=77.90.31.16:4736 , piece_id=E4BFE7Q4WTUPROYTG7IR4QLM4LL5XKCDS442KO654PB6BLEACSFQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.327366527 (445h13m56.198449562s) [noise=true , address=45.91.116.4:4939 , piece_id=DRA22PE2YDC3RYWT65F2POHKV47HEV22IGSYC5IPDVX2PQ7WDP2A, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.353346166 (445h13m56.224429211s) [noise=true , address=77.90.31.14:4661 , piece_id=TODCW4FPSPAGVDWRIASF3PKOLXUNHPWSC62QPAWKSGYFNH4UKDEQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.362861025 (445h13m56.23394406s) [noise=true , address=45.91.116.7:4010 , piece_id=KNG2IB4WHOMFXZM42EI25ML3EX6IPNWSXMDYTJUZMQYHCAW7YYWA, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.373991459 (445h13m56.245074504s) [noise=true , address=77.90.31.12:4590 , piece_id=J74VB7H77KUHJQGQLD2SDLRPYVPPKIFFQL6ZZTNYZCBTP7ZNGXAQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.463096260 (445h13m56.334179305s) [noise=true , address=45.91.116.6:4974 , piece_id=DDIMQY46RCO5U3G3ZKWIIXAV44TUGRK5YTNVAF5OQPAKM5TJ637Q, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:43:23.169031756 (445h14m23.040114801s) [noise=true , address=145.79.227.40:4049 , piece_id=FODTLUWOIDSDZA5QFJ5RZPSYHRMPFOLK5NJCRUZ7GIXC236IRDQQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:39362->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:39362->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39362->145.79.227.40:4049: use of closed network connection Dec 24 04:43:23.169171592 (445h14m23.040254637s) [noise=true , address=145.79.227.51:4410 , piece_id=2CEZTRDLXVAXFFEAEOF5SVHSOXRVB5ZTSQXYAN5GD4Y6ZU3DGV4Q, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX, address:145.79.227.51:4410): protocol: write tcp 188.166.225.170:40626->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:40626->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40626->145.79.227.51:4410: use of closed network connection Dec 24 04:43:23.169262826 (445h14m23.040345872s) [noise=true , address=145.79.227.24:4581 , piece_id=RKCKZQ7IXVKJ6WMQ7P2AQWTW5GYORBLFXAMTS7YBVY6YFK7ERCRA, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:60164->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:60164->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60164->145.79.227.24:4581: use of closed network connection Dec 24 04:43:23.169311709 (445h14m23.040394754s) [seg_pos={0 0}, upload=390839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:43:23.405890860 (445h14m23.276973895s) [upload=390839]: Done uploading segments... Dec 24 04:43:23.405935244 (445h14m23.277018279s) [upload=390839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.97973194s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.67278135s trace: https://jaeger.dev.storj.io/trace/64b51f8bd213f167 FAIL (2025-12-24 04:44:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:42:53.854141273 (445h13m53.725224318s) [upload=390839]: Starting upload Dec 24 04:42:53.854215494 (445h13m53.725298539s) [upload=390839]: Done starting upload Dec 24 04:42:53.854479075 (445h13m53.72556212s) [upload=390839]: Uploading segments... Dec 24 04:42:53.854577853 (445h13m53.725660908s) [upload=390839]: Got next segment. Inline: false Dec 24 04:42:53.854618741 (445h13m53.725701776s) [upload=390839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:42:53.854626226 (445h13m53.725709261s) [upload=390839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:42:54.098954572 (445h13m53.970037617s) [seg_pos={0 0}, upload=390839]: Begin upload segment... Dec 24 04:42:54.099000660 (445h13m53.970083695s) [seg_pos={0 0}, upload=390839]: Done begin upload segment. Dec 24 04:42:54.099029364 (445h13m53.970112409s) [noise=true , address=145.79.227.39:4013 , piece_id=7KCQCZV7IS532ADB4DLESWHRUSB5ZWIZ25HW5YNWSFXWQB4TROSQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.099140977 (445h13m53.970224022s) [noise=true , address=77.90.31.14:4661 , piece_id=TODCW4FPSPAGVDWRIASF3PKOLXUNHPWSC62QPAWKSGYFNH4UKDEQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.099586515 (445h13m53.97066955s) [noise=true , address=145.79.227.46:4602 , piece_id=XNUM44DBNEMIIEYQ66MMOZSHIWN2O5QYV4LHK2452BP5O6EOST4A, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.099863232 (445h13m53.970946267s) [noise=true , address=145.79.227.50:4373 , piece_id=EDQNNC2DIWQTGGEA7KT6BPDJGIAHT42IQZPZ7YVC3FDHTASZ3WQQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.100147754 (445h13m53.971230799s) [noise=true , address=45.91.116.4:4939 , piece_id=DRA22PE2YDC3RYWT65F2POHKV47HEV22IGSYC5IPDVX2PQ7WDP2A, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.100455209 (445h13m53.971538244s) [noise=true , address=45.91.116.7:4010 , piece_id=KNG2IB4WHOMFXZM42EI25ML3EX6IPNWSXMDYTJUZMQYHCAW7YYWA, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.100748137 (445h13m53.971831172s) [noise=true , address=145.79.227.51:4410 , piece_id=2CEZTRDLXVAXFFEAEOF5SVHSOXRVB5ZTSQXYAN5GD4Y6ZU3DGV4Q, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.101029513 (445h13m53.972112548s) [noise=true , address=45.91.116.6:4974 , piece_id=DDIMQY46RCO5U3G3ZKWIIXAV44TUGRK5YTNVAF5OQPAKM5TJ637Q, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.101365643 (445h13m53.972448678s) [noise=true , address=145.79.227.40:4049 , piece_id=FODTLUWOIDSDZA5QFJ5RZPSYHRMPFOLK5NJCRUZ7GIXC236IRDQQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.101958713 (445h13m53.973041748s) [noise=true , address=77.90.31.19:4841 , piece_id=EUONIBKTAQGXYM6CAQVZ3FJWHJQAOVFBOCCF4XYLTQI4IU566XTQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.102487159 (445h13m53.973570204s) [noise=true , address=145.79.227.32:4834 , piece_id=CIY75P6ZSW5VKRMENI7AQA3PSYBACHVKGB4XAFQYKTDBZHT7AXRQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.102824290 (445h13m53.973907325s) [noise=true , address=77.90.31.12:4590 , piece_id=J74VB7H77KUHJQGQLD2SDLRPYVPPKIFFQL6ZZTNYZCBTP7ZNGXAQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.103165290 (445h13m53.974248325s) [noise=true , address=77.90.31.16:4736 , piece_id=E4BFE7Q4WTUPROYTG7IR4QLM4LL5XKCDS442KO654PB6BLEACSFQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.104097395 (445h13m53.97518043s) [noise=true , address=145.79.227.13:4804 , piece_id=L4LQOMARXANONKYYKIUKFH3H25CJ6XFAQ2D5YVILKHXHHFKGHP7A, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.106264812 (445h13m53.977347847s) [noise=true , address=145.79.227.24:4581 , piece_id=RKCKZQ7IXVKJ6WMQ7P2AQWTW5GYORBLFXAMTS7YBVY6YFK7ERCRA, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.107835482 (445h13m53.978918517s) [noise=true , address=145.79.227.49:4339 , piece_id=OE3ELR2OHWI27NPIY6C6HI2C6DK6LIVMCBFUTJK56TCFB4ABCXQQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:55.256583228 (445h13m55.127666264s) [noise=true , address=77.90.31.19:4841 , piece_id=EUONIBKTAQGXYM6CAQVZ3FJWHJQAOVFBOCCF4XYLTQI4IU566XTQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 24 04:42:55.825208676 (445h13m55.696291711s) [upload=390839]: Next returned nil segment Dec 24 04:42:55.825231309 (445h13m55.696314354s) [upload=390839]: Waiting for error group managing segments... Dec 24 04:42:56.111380959 (445h13m55.982463994s) [noise=true , address=145.79.227.46:4602 , piece_id=XNUM44DBNEMIIEYQ66MMOZSHIWN2O5QYV4LHK2452BP5O6EOST4A, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.112285412 (445h13m55.983368457s) [noise=true , address=145.79.227.49:4339 , piece_id=OE3ELR2OHWI27NPIY6C6HI2C6DK6LIVMCBFUTJK56TCFB4ABCXQQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.112648534 (445h13m55.983731579s) [noise=true , address=145.79.227.13:4804 , piece_id=L4LQOMARXANONKYYKIUKFH3H25CJ6XFAQ2D5YVILKHXHHFKGHP7A, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.129221522 (445h13m56.000304567s) [noise=true , address=145.79.227.50:4373 , piece_id=EDQNNC2DIWQTGGEA7KT6BPDJGIAHT42IQZPZ7YVC3FDHTASZ3WQQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.135066673 (445h13m56.006149708s) [noise=true , address=145.79.227.39:4013 , piece_id=7KCQCZV7IS532ADB4DLESWHRUSB5ZWIZ25HW5YNWSFXWQB4TROSQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.136122593 (445h13m56.007205628s) [noise=true , address=145.79.227.32:4834 , piece_id=CIY75P6ZSW5VKRMENI7AQA3PSYBACHVKGB4XAFQYKTDBZHT7AXRQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.297639110 (445h13m56.168722145s) [noise=true , address=77.90.31.16:4736 , piece_id=E4BFE7Q4WTUPROYTG7IR4QLM4LL5XKCDS442KO654PB6BLEACSFQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.327366527 (445h13m56.198449562s) [noise=true , address=45.91.116.4:4939 , piece_id=DRA22PE2YDC3RYWT65F2POHKV47HEV22IGSYC5IPDVX2PQ7WDP2A, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.353346166 (445h13m56.224429211s) [noise=true , address=77.90.31.14:4661 , piece_id=TODCW4FPSPAGVDWRIASF3PKOLXUNHPWSC62QPAWKSGYFNH4UKDEQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.362861025 (445h13m56.23394406s) [noise=true , address=45.91.116.7:4010 , piece_id=KNG2IB4WHOMFXZM42EI25ML3EX6IPNWSXMDYTJUZMQYHCAW7YYWA, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.373991459 (445h13m56.245074504s) [noise=true , address=77.90.31.12:4590 , piece_id=J74VB7H77KUHJQGQLD2SDLRPYVPPKIFFQL6ZZTNYZCBTP7ZNGXAQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.463096260 (445h13m56.334179305s) [noise=true , address=45.91.116.6:4974 , piece_id=DDIMQY46RCO5U3G3ZKWIIXAV44TUGRK5YTNVAF5OQPAKM5TJ637Q, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:43:23.169031756 (445h14m23.040114801s) [noise=true , address=145.79.227.40:4049 , piece_id=FODTLUWOIDSDZA5QFJ5RZPSYHRMPFOLK5NJCRUZ7GIXC236IRDQQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:39362->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:39362->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39362->145.79.227.40:4049: use of closed network connection Dec 24 04:43:23.169171592 (445h14m23.040254637s) [noise=true , address=145.79.227.51:4410 , piece_id=2CEZTRDLXVAXFFEAEOF5SVHSOXRVB5ZTSQXYAN5GD4Y6ZU3DGV4Q, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX, address:145.79.227.51:4410): protocol: write tcp 188.166.225.170:40626->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:40626->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40626->145.79.227.51:4410: use of closed network connection Dec 24 04:43:23.169262826 (445h14m23.040345872s) [noise=true , address=145.79.227.24:4581 , piece_id=RKCKZQ7IXVKJ6WMQ7P2AQWTW5GYORBLFXAMTS7YBVY6YFK7ERCRA, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:60164->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:60164->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60164->145.79.227.24:4581: use of closed network connection Dec 24 04:43:23.169311709 (445h14m23.040394754s) [seg_pos={0 0}, upload=390839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:43:23.405890860 (445h14m23.276973895s) [upload=390839]: Done uploading segments... Dec 24 04:43:23.405935244 (445h14m23.277018279s) [upload=390839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.672810174s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.516840279s trace: https://jaeger.dev.storj.io/trace/64b51f8bd213f167 FAIL (2025-12-24 04:44:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:42:53.854141273 (445h13m53.725224318s) [upload=390839]: Starting upload Dec 24 04:42:53.854215494 (445h13m53.725298539s) [upload=390839]: Done starting upload Dec 24 04:42:53.854479075 (445h13m53.72556212s) [upload=390839]: Uploading segments... Dec 24 04:42:53.854577853 (445h13m53.725660908s) [upload=390839]: Got next segment. Inline: false Dec 24 04:42:53.854618741 (445h13m53.725701776s) [upload=390839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:42:53.854626226 (445h13m53.725709261s) [upload=390839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:42:54.098954572 (445h13m53.970037617s) [seg_pos={0 0}, upload=390839]: Begin upload segment... Dec 24 04:42:54.099000660 (445h13m53.970083695s) [seg_pos={0 0}, upload=390839]: Done begin upload segment. Dec 24 04:42:54.099029364 (445h13m53.970112409s) [noise=true , address=145.79.227.39:4013 , piece_id=7KCQCZV7IS532ADB4DLESWHRUSB5ZWIZ25HW5YNWSFXWQB4TROSQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.099140977 (445h13m53.970224022s) [noise=true , address=77.90.31.14:4661 , piece_id=TODCW4FPSPAGVDWRIASF3PKOLXUNHPWSC62QPAWKSGYFNH4UKDEQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.099586515 (445h13m53.97066955s) [noise=true , address=145.79.227.46:4602 , piece_id=XNUM44DBNEMIIEYQ66MMOZSHIWN2O5QYV4LHK2452BP5O6EOST4A, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.099863232 (445h13m53.970946267s) [noise=true , address=145.79.227.50:4373 , piece_id=EDQNNC2DIWQTGGEA7KT6BPDJGIAHT42IQZPZ7YVC3FDHTASZ3WQQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.100147754 (445h13m53.971230799s) [noise=true , address=45.91.116.4:4939 , piece_id=DRA22PE2YDC3RYWT65F2POHKV47HEV22IGSYC5IPDVX2PQ7WDP2A, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.100455209 (445h13m53.971538244s) [noise=true , address=45.91.116.7:4010 , piece_id=KNG2IB4WHOMFXZM42EI25ML3EX6IPNWSXMDYTJUZMQYHCAW7YYWA, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.100748137 (445h13m53.971831172s) [noise=true , address=145.79.227.51:4410 , piece_id=2CEZTRDLXVAXFFEAEOF5SVHSOXRVB5ZTSQXYAN5GD4Y6ZU3DGV4Q, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.101029513 (445h13m53.972112548s) [noise=true , address=45.91.116.6:4974 , piece_id=DDIMQY46RCO5U3G3ZKWIIXAV44TUGRK5YTNVAF5OQPAKM5TJ637Q, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.101365643 (445h13m53.972448678s) [noise=true , address=145.79.227.40:4049 , piece_id=FODTLUWOIDSDZA5QFJ5RZPSYHRMPFOLK5NJCRUZ7GIXC236IRDQQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.101958713 (445h13m53.973041748s) [noise=true , address=77.90.31.19:4841 , piece_id=EUONIBKTAQGXYM6CAQVZ3FJWHJQAOVFBOCCF4XYLTQI4IU566XTQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.102487159 (445h13m53.973570204s) [noise=true , address=145.79.227.32:4834 , piece_id=CIY75P6ZSW5VKRMENI7AQA3PSYBACHVKGB4XAFQYKTDBZHT7AXRQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.102824290 (445h13m53.973907325s) [noise=true , address=77.90.31.12:4590 , piece_id=J74VB7H77KUHJQGQLD2SDLRPYVPPKIFFQL6ZZTNYZCBTP7ZNGXAQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.103165290 (445h13m53.974248325s) [noise=true , address=77.90.31.16:4736 , piece_id=E4BFE7Q4WTUPROYTG7IR4QLM4LL5XKCDS442KO654PB6BLEACSFQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.104097395 (445h13m53.97518043s) [noise=true , address=145.79.227.13:4804 , piece_id=L4LQOMARXANONKYYKIUKFH3H25CJ6XFAQ2D5YVILKHXHHFKGHP7A, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.106264812 (445h13m53.977347847s) [noise=true , address=145.79.227.24:4581 , piece_id=RKCKZQ7IXVKJ6WMQ7P2AQWTW5GYORBLFXAMTS7YBVY6YFK7ERCRA, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.107835482 (445h13m53.978918517s) [noise=true , address=145.79.227.49:4339 , piece_id=OE3ELR2OHWI27NPIY6C6HI2C6DK6LIVMCBFUTJK56TCFB4ABCXQQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:55.256583228 (445h13m55.127666264s) [noise=true , address=77.90.31.19:4841 , piece_id=EUONIBKTAQGXYM6CAQVZ3FJWHJQAOVFBOCCF4XYLTQI4IU566XTQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 24 04:42:55.825208676 (445h13m55.696291711s) [upload=390839]: Next returned nil segment Dec 24 04:42:55.825231309 (445h13m55.696314354s) [upload=390839]: Waiting for error group managing segments... Dec 24 04:42:56.111380959 (445h13m55.982463994s) [noise=true , address=145.79.227.46:4602 , piece_id=XNUM44DBNEMIIEYQ66MMOZSHIWN2O5QYV4LHK2452BP5O6EOST4A, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.112285412 (445h13m55.983368457s) [noise=true , address=145.79.227.49:4339 , piece_id=OE3ELR2OHWI27NPIY6C6HI2C6DK6LIVMCBFUTJK56TCFB4ABCXQQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.112648534 (445h13m55.983731579s) [noise=true , address=145.79.227.13:4804 , piece_id=L4LQOMARXANONKYYKIUKFH3H25CJ6XFAQ2D5YVILKHXHHFKGHP7A, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.129221522 (445h13m56.000304567s) [noise=true , address=145.79.227.50:4373 , piece_id=EDQNNC2DIWQTGGEA7KT6BPDJGIAHT42IQZPZ7YVC3FDHTASZ3WQQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.135066673 (445h13m56.006149708s) [noise=true , address=145.79.227.39:4013 , piece_id=7KCQCZV7IS532ADB4DLESWHRUSB5ZWIZ25HW5YNWSFXWQB4TROSQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.136122593 (445h13m56.007205628s) [noise=true , address=145.79.227.32:4834 , piece_id=CIY75P6ZSW5VKRMENI7AQA3PSYBACHVKGB4XAFQYKTDBZHT7AXRQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.297639110 (445h13m56.168722145s) [noise=true , address=77.90.31.16:4736 , piece_id=E4BFE7Q4WTUPROYTG7IR4QLM4LL5XKCDS442KO654PB6BLEACSFQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.327366527 (445h13m56.198449562s) [noise=true , address=45.91.116.4:4939 , piece_id=DRA22PE2YDC3RYWT65F2POHKV47HEV22IGSYC5IPDVX2PQ7WDP2A, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.353346166 (445h13m56.224429211s) [noise=true , address=77.90.31.14:4661 , piece_id=TODCW4FPSPAGVDWRIASF3PKOLXUNHPWSC62QPAWKSGYFNH4UKDEQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.362861025 (445h13m56.23394406s) [noise=true , address=45.91.116.7:4010 , piece_id=KNG2IB4WHOMFXZM42EI25ML3EX6IPNWSXMDYTJUZMQYHCAW7YYWA, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.373991459 (445h13m56.245074504s) [noise=true , address=77.90.31.12:4590 , piece_id=J74VB7H77KUHJQGQLD2SDLRPYVPPKIFFQL6ZZTNYZCBTP7ZNGXAQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.463096260 (445h13m56.334179305s) [noise=true , address=45.91.116.6:4974 , piece_id=DDIMQY46RCO5U3G3ZKWIIXAV44TUGRK5YTNVAF5OQPAKM5TJ637Q, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:43:23.169031756 (445h14m23.040114801s) [noise=true , address=145.79.227.40:4049 , piece_id=FODTLUWOIDSDZA5QFJ5RZPSYHRMPFOLK5NJCRUZ7GIXC236IRDQQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:39362->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:39362->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39362->145.79.227.40:4049: use of closed network connection Dec 24 04:43:23.169171592 (445h14m23.040254637s) [noise=true , address=145.79.227.51:4410 , piece_id=2CEZTRDLXVAXFFEAEOF5SVHSOXRVB5ZTSQXYAN5GD4Y6ZU3DGV4Q, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX, address:145.79.227.51:4410): protocol: write tcp 188.166.225.170:40626->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:40626->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40626->145.79.227.51:4410: use of closed network connection Dec 24 04:43:23.169262826 (445h14m23.040345872s) [noise=true , address=145.79.227.24:4581 , piece_id=RKCKZQ7IXVKJ6WMQ7P2AQWTW5GYORBLFXAMTS7YBVY6YFK7ERCRA, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:60164->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:60164->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60164->145.79.227.24:4581: use of closed network connection Dec 24 04:43:23.169311709 (445h14m23.040394754s) [seg_pos={0 0}, upload=390839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:43:23.405890860 (445h14m23.276973895s) [upload=390839]: Done uploading segments... Dec 24 04:43:23.405935244 (445h14m23.277018279s) [upload=390839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.516923448s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 21.184042662s trace: https://jaeger.dev.storj.io/trace/64b51f8bd213f167 FAIL (2025-12-24 04:43:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 24 04:42:53.854141273 (445h13m53.725224318s) [upload=390839]: Starting upload Dec 24 04:42:53.854215494 (445h13m53.725298539s) [upload=390839]: Done starting upload Dec 24 04:42:53.854479075 (445h13m53.72556212s) [upload=390839]: Uploading segments... Dec 24 04:42:53.854577853 (445h13m53.725660908s) [upload=390839]: Got next segment. Inline: false Dec 24 04:42:53.854618741 (445h13m53.725701776s) [upload=390839]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 24 04:42:53.854626226 (445h13m53.725709261s) [upload=390839]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 24 04:42:54.098954572 (445h13m53.970037617s) [seg_pos={0 0}, upload=390839]: Begin upload segment... Dec 24 04:42:54.099000660 (445h13m53.970083695s) [seg_pos={0 0}, upload=390839]: Done begin upload segment. Dec 24 04:42:54.099029364 (445h13m53.970112409s) [noise=true , address=145.79.227.39:4013 , piece_id=7KCQCZV7IS532ADB4DLESWHRUSB5ZWIZ25HW5YNWSFXWQB4TROSQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.099140977 (445h13m53.970224022s) [noise=true , address=77.90.31.14:4661 , piece_id=TODCW4FPSPAGVDWRIASF3PKOLXUNHPWSC62QPAWKSGYFNH4UKDEQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.099586515 (445h13m53.97066955s) [noise=true , address=145.79.227.46:4602 , piece_id=XNUM44DBNEMIIEYQ66MMOZSHIWN2O5QYV4LHK2452BP5O6EOST4A, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.099863232 (445h13m53.970946267s) [noise=true , address=145.79.227.50:4373 , piece_id=EDQNNC2DIWQTGGEA7KT6BPDJGIAHT42IQZPZ7YVC3FDHTASZ3WQQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.100147754 (445h13m53.971230799s) [noise=true , address=45.91.116.4:4939 , piece_id=DRA22PE2YDC3RYWT65F2POHKV47HEV22IGSYC5IPDVX2PQ7WDP2A, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.100455209 (445h13m53.971538244s) [noise=true , address=45.91.116.7:4010 , piece_id=KNG2IB4WHOMFXZM42EI25ML3EX6IPNWSXMDYTJUZMQYHCAW7YYWA, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.100748137 (445h13m53.971831172s) [noise=true , address=145.79.227.51:4410 , piece_id=2CEZTRDLXVAXFFEAEOF5SVHSOXRVB5ZTSQXYAN5GD4Y6ZU3DGV4Q, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.101029513 (445h13m53.972112548s) [noise=true , address=45.91.116.6:4974 , piece_id=DDIMQY46RCO5U3G3ZKWIIXAV44TUGRK5YTNVAF5OQPAKM5TJ637Q, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.101365643 (445h13m53.972448678s) [noise=true , address=145.79.227.40:4049 , piece_id=FODTLUWOIDSDZA5QFJ5RZPSYHRMPFOLK5NJCRUZ7GIXC236IRDQQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.101958713 (445h13m53.973041748s) [noise=true , address=77.90.31.19:4841 , piece_id=EUONIBKTAQGXYM6CAQVZ3FJWHJQAOVFBOCCF4XYLTQI4IU566XTQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.102487159 (445h13m53.973570204s) [noise=true , address=145.79.227.32:4834 , piece_id=CIY75P6ZSW5VKRMENI7AQA3PSYBACHVKGB4XAFQYKTDBZHT7AXRQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.102824290 (445h13m53.973907325s) [noise=true , address=77.90.31.12:4590 , piece_id=J74VB7H77KUHJQGQLD2SDLRPYVPPKIFFQL6ZZTNYZCBTP7ZNGXAQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.103165290 (445h13m53.974248325s) [noise=true , address=77.90.31.16:4736 , piece_id=E4BFE7Q4WTUPROYTG7IR4QLM4LL5XKCDS442KO654PB6BLEACSFQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.104097395 (445h13m53.97518043s) [noise=true , address=145.79.227.13:4804 , piece_id=L4LQOMARXANONKYYKIUKFH3H25CJ6XFAQ2D5YVILKHXHHFKGHP7A, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.106264812 (445h13m53.977347847s) [noise=true , address=145.79.227.24:4581 , piece_id=RKCKZQ7IXVKJ6WMQ7P2AQWTW5GYORBLFXAMTS7YBVY6YFK7ERCRA, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:54.107835482 (445h13m53.978918517s) [noise=true , address=145.79.227.49:4339 , piece_id=OE3ELR2OHWI27NPIY6C6HI2C6DK6LIVMCBFUTJK56TCFB4ABCXQQ, seg_pos={0 0}, upload=390839]: Uploading piece... Dec 24 04:42:55.256583228 (445h13m55.127666264s) [noise=true , address=77.90.31.19:4841 , piece_id=EUONIBKTAQGXYM6CAQVZ3FJWHJQAOVFBOCCF4XYLTQI4IU566XTQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Dec 24 04:42:55.825208676 (445h13m55.696291711s) [upload=390839]: Next returned nil segment Dec 24 04:42:55.825231309 (445h13m55.696314354s) [upload=390839]: Waiting for error group managing segments... Dec 24 04:42:56.111380959 (445h13m55.982463994s) [noise=true , address=145.79.227.46:4602 , piece_id=XNUM44DBNEMIIEYQ66MMOZSHIWN2O5QYV4LHK2452BP5O6EOST4A, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.112285412 (445h13m55.983368457s) [noise=true , address=145.79.227.49:4339 , piece_id=OE3ELR2OHWI27NPIY6C6HI2C6DK6LIVMCBFUTJK56TCFB4ABCXQQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.112648534 (445h13m55.983731579s) [noise=true , address=145.79.227.13:4804 , piece_id=L4LQOMARXANONKYYKIUKFH3H25CJ6XFAQ2D5YVILKHXHHFKGHP7A, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.129221522 (445h13m56.000304567s) [noise=true , address=145.79.227.50:4373 , piece_id=EDQNNC2DIWQTGGEA7KT6BPDJGIAHT42IQZPZ7YVC3FDHTASZ3WQQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.135066673 (445h13m56.006149708s) [noise=true , address=145.79.227.39:4013 , piece_id=7KCQCZV7IS532ADB4DLESWHRUSB5ZWIZ25HW5YNWSFXWQB4TROSQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.136122593 (445h13m56.007205628s) [noise=true , address=145.79.227.32:4834 , piece_id=CIY75P6ZSW5VKRMENI7AQA3PSYBACHVKGB4XAFQYKTDBZHT7AXRQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.297639110 (445h13m56.168722145s) [noise=true , address=77.90.31.16:4736 , piece_id=E4BFE7Q4WTUPROYTG7IR4QLM4LL5XKCDS442KO654PB6BLEACSFQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.327366527 (445h13m56.198449562s) [noise=true , address=45.91.116.4:4939 , piece_id=DRA22PE2YDC3RYWT65F2POHKV47HEV22IGSYC5IPDVX2PQ7WDP2A, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.353346166 (445h13m56.224429211s) [noise=true , address=77.90.31.14:4661 , piece_id=TODCW4FPSPAGVDWRIASF3PKOLXUNHPWSC62QPAWKSGYFNH4UKDEQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.362861025 (445h13m56.23394406s) [noise=true , address=45.91.116.7:4010 , piece_id=KNG2IB4WHOMFXZM42EI25ML3EX6IPNWSXMDYTJUZMQYHCAW7YYWA, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.373991459 (445h13m56.245074504s) [noise=true , address=77.90.31.12:4590 , piece_id=J74VB7H77KUHJQGQLD2SDLRPYVPPKIFFQL6ZZTNYZCBTP7ZNGXAQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:42:56.463096260 (445h13m56.334179305s) [noise=true , address=45.91.116.6:4974 , piece_id=DDIMQY46RCO5U3G3ZKWIIXAV44TUGRK5YTNVAF5OQPAKM5TJ637Q, seg_pos={0 0}, upload=390839]: Done uploading piece. err: <nil> Dec 24 04:43:23.169031756 (445h14m23.040114801s) [noise=true , address=145.79.227.40:4049 , piece_id=FODTLUWOIDSDZA5QFJ5RZPSYHRMPFOLK5NJCRUZ7GIXC236IRDQQ, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:39362->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:39362->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39362->145.79.227.40:4049: use of closed network connection Dec 24 04:43:23.169171592 (445h14m23.040254637s) [noise=true , address=145.79.227.51:4410 , piece_id=2CEZTRDLXVAXFFEAEOF5SVHSOXRVB5ZTSQXYAN5GD4Y6ZU3DGV4Q, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX, address:145.79.227.51:4410): protocol: write tcp 188.166.225.170:40626->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:40626->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40626->145.79.227.51:4410: use of closed network connection Dec 24 04:43:23.169262826 (445h14m23.040345872s) [noise=true , address=145.79.227.24:4581 , piece_id=RKCKZQ7IXVKJ6WMQ7P2AQWTW5GYORBLFXAMTS7YBVY6YFK7ERCRA, seg_pos={0 0}, upload=390839]: Done uploading piece. err: ecclient: upload failed (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1, address:145.79.227.24:4581): protocol: write tcp 188.166.225.170:60164->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:60164->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60164->145.79.227.24:4581: use of closed network connection Dec 24 04:43:23.169311709 (445h14m23.040394754s) [seg_pos={0 0}, upload=390839]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 24 04:43:23.405890860 (445h14m23.276973895s) [upload=390839]: Done uploading segments... Dec 24 04:43:23.405935244 (445h14m23.277018279s) [upload=390839]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 21.184058042s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)